Orthopedic Fixation Control And Visualization

ABSTRACT

A guided frame matching technique may be employed in which a graphical projection of a fixator is used to guide and assist a user in identifying fixator elements within an image. Upon being overlaid on an image, the graphical projection of fixator may be manipulated to align with, and overlay, fixator elements displayed within the image. Additionally, a three-dimensional overview of an imaging scene of the fixator may be generated. The three-dimensional overview may include a three-dimensional graphical model that displays representations of images of the fixator attached to anatomical structure segments as well as representations of imaging sources, reference locations, fixator elements, and anatomical structure segments. When the images are not truly orthogonal, a modified second image representation may be displayed to represent a second image that is truly orthogonal to a first image.

BACKGROUND

Techniques used to treat fractures and/or deformities of anatomicalstructures, such as bones, can include the use of external fixators,such as hexapods and other fixation frames, which are surgically mountedto anatomical structure segments on opposed sides of a fracture site. Apair of radiographic images is taken of the fixator and anatomicalstructure segments at the fracture site. Data from the images is thenmanipulated to construct a three-dimensional representation of thefixator and the anatomical structures segments that can be used indeveloping a treatment plan, which may for example comprise realigningthe anatomical structure segments through adjustments to the fixator.

Existing techniques for controlling fixator manipulation may, however,involve a number of limitations that may introduce inefficiency,complication, and unreliability. For example, some conventionaltechniques may rely on a surgeon or other user to indicate locations ofcertain fixator elements, such as struts, within images that aredisplayed in a graphical user interface of a computer. However, it mayoften be difficult for the user to identify and mark positions of thestruts and other fixator elements within the images. In particular,depending upon the location and orientation from which an image iscaptured, struts and other fixator elements may be not be identifiedeasily, such as because they may wholly or partially overlap one anotheror may otherwise be obscured within the images. This may make itcumbersome for the user to identify the fixator elements, therebyincreasing time required to identify the elements, increasing theprobability of errors, and reducing the reliability of the calculations.As another example, some conventional techniques may be limited withrespect to the ability to visualize a complete three-dimensional fixatorimaging scene for the user. Additionally, when images arenon-orthogonal, the user may be unable to efficiently viewrepresentations of various corrections, such as corrections to theanatomical structure angulations, calculated by the softwarecorresponding to true orthogonal images. Thus, insufficient feedback maybe provided to the user, thereby resulting in additional errors. Thismay reduce the reliability of the treatment plan, possibly resulting inimproper alignment of anatomical structures segments during the healingprocess, compromised union between the anatomical structure segments,necessitating additional rounds of radiographic imaging to facilitatealignment corrections, or even necessitating additional surgicalprocedures.

SUMMARY

Techniques for orthopedic fixation control and visualization, forexample for correction of a deformity of an anatomical structure, suchas a bone, are described herein. In particular, in some examples, afixation apparatus may be attached to first and second anatomicalstructure segments. Images, such as x-rays, of the fixation apparatusand the attached anatomical structure segments may then be captured fromdifferent orientations with respect to the fixation apparatus.

In some examples, various manipulations to the fixation apparatus forcorrection of the anatomical structure deformity may be determined basedon positions and orientations of the anatomical structure segments inthree-dimensional space. Also, in some examples, the positions andorientations of the anatomical structure segments in three-dimensionalspace may be determined based on the images. In particular, in somecases, the positions and orientations of the anatomical structuresegments in three-dimensional space may be determined by having asurgeon or other user indicate locations of various fixator elements andanatomical structures within the images. However, as described above, itmay often be difficult for the user to identify and mark positions ofcertain fixator elements, such as struts, within the images. Inparticular, depending upon the location and orientation from which animage is captured, struts and other fixator elements may be not beidentified easily, such as because they may wholly or partially overlapone another or may otherwise be obscured within the images. For example,in some cases, it may often be more difficult to identify struts withina lateral image than to identify the struts within an anterior image,such as because the struts may often overlap one another when viewedwithin the lateral image. This may make it cumbersome for the user toidentify the fixator elements, thereby increasing time required toidentify the elements, increasing the probability of errors, andreducing the reliability of the calculations.

In some examples, to alleviate the above and other problems, a guidedframe matching technique may be employed in which a graphical projectionof the fixator is used to guide and assist the user in identifyingfixator elements within an image. In particular, locations of fixatorelements, such as rings and/or struts of the fixator, may be identifiedby a user within a first image, for example an anterior image or otherimage in which the fixator elements are easily identifiable. A graphicalprojection of the fixator, such as including graphical representationsof the rings and/or struts, may then be generated based at least in parton the identified locations of the fixator elements in the first image.The graphical projection of the fixator may then be overlaid upon asecond image, such as a lateral image or other image in which thefixator elements are not easily identifiable. The graphical projectionof the fixator may then be used within the second image as a guide, suchas to assist the user in identifying locations of the fixator elementswithin the second image. In some examples, the graphical projection ofthe fixator may be rotated relative to the locations of the fixatorelements in the first image. In particular, the graphical projection ofthe fixator may be rotated based at least in part on an angle of imageplanes of the first and the second images with respect to one another.For example, if the first image is an anterior image, and the secondimage is a lateral image at an angle of ninety degrees to the anteriorimage, then the graphical projection of the fixator, upon being overlaidonto the second image, may be rotated ninety degrees relative to theidentified locations of the fixator elements in the first image. Thegraphical projection of the fixator may also be modified within thesecond image by a user, for example by moving, resizing and/or furtherrotating the graphical projection of the fixator, such as to betteralign with the locations of the fixator elements in the second image.Upon being overlaid on the second image, the graphical projection offixator may serve as a guide to assist the user in identifying fixatorelements within the second image, such as by manipulating the graphicalprojection of the fixator to align with, and overlay, fixator elementsdisplayed within the second image.

Another technique for improving accuracy and reliability of input valuesand resulting calculations is disclosed herein that provides athree-dimensional overview of an imaging scene of the fixator. Thethree-dimensional overview may be used to provide feedback and visualconfirmation to help ensure that the calculated positions andorientations of anatomical structures is reliable and correct. Inparticular, the three-dimensional overview may include athree-dimensional graphical model that is displayed in a graphical userinterface of a computing system. The three-dimensional graphical modelmay include a first image representation that represents a first imageand may include a second image representation that represents a secondimage. The first image representation and the second imagerepresentation may be displayed at an angle that is the same as theangle between the image planes of the first and second images. Thethree-dimensional model may also include graphical representations ofvirtual locations corresponding to respective first and second imagingsources of the first and the second image. A first virtual line mayconnect a first virtual location corresponding to the first imagingsource to a reference location on the first image representation. Asecond virtual line may connect a second virtual location correspondingto the second imaging source to the same reference location on thesecond image representation. In some examples, the reference locationmay be a reference point of the first or the second anatomical structuresegment, such as an endpoint of the anatomical structure segment. Thethree-dimensional graphical model may display a first graphicalrepresentation associated with a shortest distance (e.g., intersection)between the first virtual line and the second virtual line, and thisfirst graphical representation may represent a physical location of thereference location in three-dimensional space. The three-dimensionalgraphical model may also display graphical representations of fixatorelements (rings, struts, etc.) at virtual locations that representphysical locations of the fixator elements in the three-dimensionalspace. The three-dimensional graphical model may also display graphicalrepresentations of the anatomical structure segments at virtuallocations that represent physical locations of the anatomical structuresegments in the three-dimensional space. The three-dimensional graphicalmodel may be zoomed, panned, and rotated in any combination of one ormore directions by a user.

In some examples, when the first and the second image planes arenon-orthogonal, the three-dimensional graphical model may display amodified second image representation that is orthogonal to the firstimage representation. The modified second image representation mayrepresent a modified second image having an image plane that is trulyorthogonal to the image plane of the first image. In some examples, thesoftware may calculate the angle of the anatomical structure segmentsthat would be displayed in the second image if the second image weretruly orthogonal to the first image. The software may then display, inthe modified second image representation, a modified second image inwhich the anatomical structure segments are displayed with thecalculated angle with respect to one another. In this way, the softwaremay demonstrate to the user how measured anatomical structure deformityvalues from a second image that is not truly orthogonal to the firstimage may be adjusted to corrected anatomical structure deformity valuesfor truly orthogonal images, and the software provide a guidance tovalidate the corrected values for use in the deformity calculations.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The foregoing summary, as well as the following detailed description ofthe preferred embodiments of the application, will be better understoodwhen read in conjunction with the appended drawings. For the purposes ofillustrating the methods and/or techniques of orthopedic fixation withimagery analysis, there are shown in the drawings preferred embodiments.It should be understood, however, that the instant application is notlimited to the precise arrangements and/or instrumentalities illustratedin the drawings, in which:

FIG. 1 is a perspective view of a fixation assembly positioned forimaging in accordance with an embodiment;

FIG. 2 is a perspective view of an example imaging process of thefixation assembly illustrated in FIG. 1;

FIGS. 3A and 3B are flow diagrams illustrating an example process forcontrolling manipulation of a fixation apparatus to correct ananatomical structure deformity;

FIG. 4 is a screen shot of an example interface for selecting aPerspective Frame Matching (PFM) technique;

FIG. 5 is a screen shot of an example configuration information entryinterface for the PFM technique;

FIG. 6 is a screen shot of an example first image information entryinterface for the PFM technique;

FIG. 7 is a screen shot of an example close-up assist interface for thePFM technique;

FIGS. 8A-8H are screen shots of an example second image informationentry interface for the PFM technique;

FIG. 9 is a screen shot of example deformity parameter interface for thePFM technique;

FIG. 10 is a screen shot of an example mounting parameter interface forthe PFM technique;

FIG. 11 is a screen shot of a first example treatment plan interface forthe PFM technique;

FIG. 12 is a screen shot of a second example treatment plan interfacefor the PFM technique;

FIG. 13 is a screen shot of a third example treatment plan interface forthe PFM technique;

FIG. 14 is a flow diagram illustrating an example process for providinga graphical projection of a fixator using a guided frame matchingtechnique;

FIG. 15A is a diagram illustrating example images of first and secondanatomical structure segments and a fixator attached thereto;

FIG. 15B is a diagram illustrating example first and second images of afixator in which locations of fixator elements are indicated in thefirst image but not the second image;

FIG. 16 is a diagram illustrating example graphical projection of afixator overlaid upon an image;

FIG. 17A is a diagram illustrating an example graphical projection of afixator that is manipulated by a user;

FIG. 17B is a diagram illustrating example first and second images of afixator in which locations of fixator elements are indicated in both thefirst image and the second image;

FIG. 18 is a flow diagram illustrating an example process for generatinga three-dimensional overview of an imaging scene of a fixator;

FIG. 19 is a diagram illustrating an example three-dimensional graphicalmodel of the imaging scene of the fixator;

FIG. 20 is a diagram illustrating an example three-dimensional graphicalmodel including graphical representations of fixator rings;

FIG. 21 is a diagram illustrating an example three-dimensional graphicalmodel including graphical representations of fixator struts andanatomical structure segments;

FIG. 22 is a diagram illustrating an example three-dimensional graphicalmodel from an anteroposterior perspective;

FIG. 23 is a diagram illustrating an example three-dimensional graphicalmodel from a lateral perspective;

FIG. 24 is a diagram illustrating an example rotated three-dimensionalgraphical model;

FIG. 25 is a diagram illustrating an example modified second imagerepresentation; and

FIG. 26 is a block diagram of an example computing device for use inaccordance with the present disclosure.

DETAILED DESCRIPTION

For convenience, the same or equivalent elements in the variousembodiments illustrated in the drawings have been identified with thesame reference numerals. Certain terminology is used in the followingdescription for convenience only and is not limiting. The words “right”,“left”, “top” and “bottom” designate directions in the drawings to whichreference is made. The words “inward”, “inwardly”, “outward”, and“outwardly” refer to directions toward and away from, respectively, thegeometric center of the device and designated parts thereof Theterminology intended to be non-limiting includes the above-listed words,derivatives thereof and words of similar import.

Referring initially to FIG. 1, bodily tissues, for instance first andsecond anatomical structure segments 102, 104, can be aligned and/ororiented to promote union or other healing between the bodily tissues.Anatomical structures may include, for example, anatomical tissue andartificial anatomical implants. Anatomical tissue may include, forexample, bone or other tissue in the body. The alignment and/ororientation of the bodily tissues can be achieved by connecting thebodily tissues to an adjustable fixation apparatus, such as orthopedicfixator 100. The orthopedic fixator can comprise an external fixationapparatus that includes a plurality of discrete fixator members thatremain external to the patient's body, but that are attached torespective discreet bodily tissues, for example with minimally invasiveattachment members. A fixation apparatus may include, for example, adistraction osteogenesis ring system, a hexapod, or a Taylor spatialframe. By adjusting the spatial positioning of the fixator members withrespect to each other, the respective bodily tissues attached theretocan be reoriented and/or otherwise brought into alignment with eachother, for example to promote union between the bodily tissues duringthe healing process. The use of external orthopedic fixators incombination with the imagery analysis and positioning techniquesdescribed herein can be advantageous in applications where directmeasurement and manipulation of the bodily tissues is not possible,where limited or minimally invasive access to the bodily tissues isdesired, or the like. Some examples of orthopedic fixators and their usefor correcting deformities of anatomical structure segments, as well astechniques for performing imagery analysis on the fixators andanatomical structure segments are described in U.S. Pat. No. 9,642,649,entitled “ORTHOPEDIC FIXATION WITH IMAGERY ANALYSIS,” issued on May 9,2017, the entirety of which is hereby incorporated by reference.

The fixator members can be connected to each other via adjustmentmembers, the adjustment members configured to facilitate the spatialrepositioning of the fixator members with respect to each other. Forexample, in the illustrated embodiment, the orthopedic fixator 100comprises a pair of fixator members in the form of an upper fixator ring106 and a lower fixator ring 108. The fixator rings 106, 108 can beconstructed the same or differently. For instance, the fixator rings106, 108 can have diameters that are the same or different. Similarly,the fixator rings 106, 108 can be constructed with varying crosssectional diameters, thicknesses, etc. It should be appreciated that thefixator members of the orthopedic fixator 100 are not limited to theillustrated upper and lower fixator rings 106, 108, and that theorthopedic fixator 100 can be alternatively constructed. For example,additional fixator rings can be provided and interconnected with thefixator ring 106 and/or 108. It should further be appreciated that thegeometries of the fixator members are not limited to rings, and that atleast one, such as all of the fixator members can be alternativelyconstructed using any other suitable geometry.

The first and second anatomical structure segments 102, 104 can berigidly attached to the upper and lower fixator rings 106, 108,respectively, with attachment members that can be mounted to the fixatorrings 106, 108. For example, in the illustrated embodiment, attachmentmembers are provided in the form of attachment rods 110 and attachmentwires 112.

The rods 110 and the wires 112 extend between proximal ends attached tomounting members 114 that are mounted to the fixator rings 106, 108, andopposed distal ends that are inserted into or otherwise secured to theanatomical structure segments 102, 104. The mounting members 114 can beremovably mounted to the fixator rings 106, 108 at predefined pointsalong the peripheries of the fixator rings 106, 108, for example bydisposing them into threaded apertures defined by the fixator rings.With respect to each fixator ring 106, 108, the mounting members 114 canbe mounted to the upper surface of the ring, the lower surface of thering, or any combination thereof. It should be appreciated that theattachment members are not limited to the configuration of theillustrated embodiment. For example, any number of attachment members,such as the illustrated rods 110 and wires 112 and any others, can beused to secure the anatomical structure segments to respective fixatormembers as desired. It should further be appreciated that one or more ofthe attachment members, for instance the rods 110 and/or wires 112, canbe alternatively configured to mount directly to the fixator rings 106,108, without utilizing mounting members 114.

The upper and lower fixator rings 106, 108 can be connected to eachother by at least one, such as a plurality of adjustment members. Atleast one, such as all, of the adjustment members can be configured toallow the spatial positioning of the fixator rings with respect to eachother to be adjusted. For example, in the illustrated embodiment, theupper and lower fixator rings 106, 108 are connected to each other witha plurality of adjustment members provided in the form of adjustablelength struts 116. It should be appreciated that the construction of theorthopedic fixator 100 is not limited to the six struts 116 of theillustrated embodiment, and that more or fewer struts can be used asdesired.

Each of the adjustable length struts 116 can comprise opposed upper andlower strut arms 118, 120. Each of the upper and lower strut arms 118,120 have proximal ends disposed in a coupling member, or sleeve 122, andopposed distal ends that are coupled to universal joints 124 mounted tothe upper and lower fixator rings 106, 108, respectively. The universaljoints of the illustrated embodiment are disposed in pairs spaced evenlyaround the peripheries of the upper and lower fixator rings 106, 108,but can be alternatively placed in any other locations on the fixatorrings as desired.

The proximal ends of the upper and lower strut arms 118, 120 of eachstrut 116 can have threads defined thereon that are configured to bereceived by complementary threads defined in the sleeve 122, such thatwhen the proximal ends of the upper and lower strut arms 118, 120 of astrut 116 are received in a respective sleeve 122, rotation of thesleeve 122 causes the upper and lower strut arms 118, 120 to translatewithin the sleeve 122, thus causing the strut 116 to be elongated orshortened, depending on the direction of rotation. Thus, the length ofeach strut 116 can be independently adjusted with respect to theremaining struts. It should be appreciated that the adjustment membersare not limited to the length adjustable struts 116 of the illustratedembodiment, and that the adjustment members can be alternativelyconstructed as desired, for example using one or more alternativegeometries, alternative length adjustment mechanisms, and the like.

The adjustable length struts 116 and the universal joints 124 by whichthey are mounted to the upper and lower fixator rings 106, 108, allowthe orthopedic fixator 100 to function much like a Stewart platform, andmore specifically like a distraction osteogenesis ring system, ahexapod, or a Taylor spatial frame. That is, by making lengthadjustments to the struts 116, the spatial positioning of the upper andlower fixator rings 106, 108, and thus the anatomical structure segments102, 104 can be altered. For example, in the illustrated embodiment thefirst anatomical structure segment 102 is attached to the upper fixatorring 106 and the second anatomical structure segment 104 is attached tothe lower fixator ring 108. It should be appreciated that attachment ofthe first and second anatomical structure segments 102, 104 to the upperand lower fixator rings 106, 108 is not limited to the illustratedembodiment (e.g., where the central longitudinal axes L1, L2 of thefirst and second anatomical structure segments 102, 104 aresubstantially perpendicular to the respective planes of the upper andlower fixator rings 106, 108), and that a surgeon has completeflexibility in aligning the first and second anatomical structuresegments 102, 104 within the upper and lower fixator rings 106, 108 whenconfiguring the orthopedic fixator 100.

By varying the length of one or more of the struts 116, the upper andlower fixator rings 106, 108, and thus the anatomical structure segments102 and 104 can be repositioned with respect to each other such thattheir respective longitudinal axes L1, L2 are substantially aligned witheach other, and such that their respective fractured ends 103, 105 abuteach other, so as to promote union during the healing process. It shouldbe appreciated that adjustment of the struts 116 is not limited to thelength adjustments as described herein, and that the struts 116 can bedifferently adjusted as desired. It should further be appreciated thatadjusting the positions of the fixator members is not limited toadjusting the lengths of the length adjustable struts 116, and that thepositioning of the fixator members with respect to each other can bealternatively adjusted, for example in accordance the type and/or numberof adjustment members connected to the fixation apparatus.

Repositioning of the fixator members of an orthopedic fixationapparatus, such as orthopedic fixator 100, can be used to correctdisplacements of angulation, translation, rotation, or any combinationthereof, within bodily tissues. A fixation apparatus, such as orthopedicfixator 100, utilized with the techniques described herein, can correcta plurality of such displacement defects individually or simultaneously.However, it should be appreciated that the fixation apparatus is notlimited to the illustrated orthopedic fixator 100, and that the fixationapparatus can be alternatively constructed as desired. For example, thefixation apparatus can include additional fixation members, can includefixation members having alternative geometries, can include more orfewer adjustment members, can include alternatively constructedadjustment members, or any combination thereof

Referring now to FIG. 2, an example imaging of a fixation apparatus willnow be described in detail. The images can be captured using the same ordifferent imaging techniques. For example, the images can be acquiredusing x-ray imaging, computer tomography, magnetic resonance imaging,ultrasound, infrared imaging, photography, fluoroscopy, visual spectrumimaging, or any combination thereof

The images can be captured from any position and/or orientation withrespect to each other and with respect to the fixator 100 and theanatomical structure segments 102, 104. In other words, there is norequirement that the captured images be orthogonal with respect to eachother or aligned with anatomical axes of the patient, thereby providinga surgeon with near complete flexibility in positioning the imagers 130.Preferably, the images 126, 128 are captured from different directions,or orientations, such that the images do not overlap. For example, inthe illustrated embodiment, the image planes of the pair of images 126,128 are not perpendicular with respect to each other. In other words,the angle a between the image planes of the images 126, 128 is not equalto 90 degrees, such that the images 126, 128 are non-orthogonal withrespect to each other. Preferably, at least two images are taken,although capturing additional images may increase the accuracy of themethod.

The images 126, 128 can be captured using one or more imaging sources,or imagers, for instance the x-ray imagers 130 and/or correspondingimage capturing devices 127, 129. The images 126, 128 can be x-rayimages captured by a single repositionable x-ray imager 130, or can becaptured by separately positioned imagers 130. Preferably, the positionof the image capturing devices 127, 129 and/or the imagers 130 withrespect to the space origin 135 of the three-dimensional space,described in more detail below, are known. The imagers 130 can bemanually positioned and/or oriented under the control of a surgeon,automatically positioned, for instance by a software assisted imager, orany combination thereof. The fixator 100 may also have a respectivefixator origin 145.

Referring now to FIGS. 3A and 3B, an example process for controllingmanipulation of a fixation apparatus including rings and struts tocorrect an anatomical structure deformity of first and second anatomicalstructure segments will now be described in detail. In particular, atoperation 310, first and second anatomical structure segments areattached to a fixation apparatus, for example as shown in FIG. 1 anddescribed in detail above. At operation, 312, first and second images ofthe fixation apparatus and the attached first and second anatomicalstructure segments are captured, for example as shown in FIG. 2 anddescribed in detail above.

The remaining operations of the process of FIGS. 3A and 3B (e.g.,operations 314-342) will now be described in association with atreatment technique referred to hereinafter as Perspective FrameMatching, in which images, such as post-operative x-rays, may be usedalong with a frame to generate deformity and mounting parameters for astrut adjustment plan. For example, referring now to FIG. 4, an exampletreatment planning technique selection interface 400-A is shown. In theexample of FIG. 4, the user has selected option 401 in order to use thePerspective Frame Matching (PFM) technique, which will now be describedin detail with reference to FIGS. 5-13.

Referring back to FIG. 3A, at operation 314, configuration informationassociated with a fixation apparatus is received, for example using oneor more graphical user interfaces of a computing system. In someexamples, the configuration information may include one or moregeometric characteristics (e.g., size, length, diameter, etc.) of one ormore elements of the fixation apparatus, such as struts, hinges, rings,and others. In some examples, the configuration information may includeinformation such as ring types (e.g., full ring, foot plate, etc.),indications of mounting points (e.g., ring holes) used for strutmounting, and other information. In some examples, the configurationinformation may also include information about marker elements, forexample that are mounted to components of the fixation apparatus, suchas struts, hinges, and rings. Referring now to FIG. 5, an exampleconfiguration information entry interface 500 is shown. As shown,interface 500 includes ring type indicators 501 and 502, which, in thisexample, are drop down menus that may be used to select ring types forthe proximal and distal rings, respectively. Indicators 501 and 502 areset to the “Full” option to indicate that the proximal and distal ringsare full rings. Interface 500 also includes diameter indicators 503 and504, which, in this example, are drop down menus that may be used toselect diameters or lengths for the proximal and distal rings,respectively.

The interface 500 also includes controls for entry of strut information.In particular, interface 500 includes six drop down menus 512 may eachbe used to indicate a size of a respective strut. Global strut sizeindicator 511 may also be used to globally select a size for all sixstruts. Length selectors 513 may be each be used to select a length of arespective strut. Length indicators 514 may be each be used to provide avisual representation of the lengths of the respective struts. It isnoted that the length indicators 514 do not necessarily depict theactual exact length of each strut, but rather represent the comparativelengths of the struts with respect to one another.

Save and Update button 516 may be selected to save and update theconfiguration information values shown in interface 500. In someexamples, selection of button 516 may cause interface 500 to displayand/or update a graphical representation 520 of the fixation apparatusgenerated based, at least in part, on the entered configurationinformation. The graphical representation 520 may be displayed using oneor more graphical user interfaces of a computing system. As shown,graphical representation 520 includes six struts that may be color-codedin multiple colors for easy identification. For example, in some cases,each of the struts (or at least two of the struts) may be shown indifferent colors with respect to one another. The struts in graphicalrepresentation 520 may have sizes, lengths, mounting points, and otherfeatures corresponding to entered configuration information. Graphicalrepresentation 520 also depicts the fixator rings, which may havediameters/lengths, ring types, and other features corresponding toentered configuration information. Graphical representation 520 may, forexample, improve efficiency and reliability by providing the user with avisual confirmation of information entered into interface 500, forexample to allow fast and easy identification of errors or otherproblems.

At operation 316, images of the fixation apparatus and the first andsecond anatomical structure segments attached thereto are displayed, forexample using one or more graphical user interfaces of a computingsystem. The displayed images may include images that were captured atoperation 312, such as using x-ray imaging, computer tomography,magnetic resonance imaging, ultrasound, infrared imaging, photography,fluoroscopy, visual spectrum imaging, or any combination thereof.Techniques for acquiring images of the fixation apparatus and the firstand second anatomical structure segments are described in detail aboveand are not repeated here. As set forth above, the acquired anddisplayed images need not necessarily be orthogonal to one another.Referring now to FIG. 6, an example first image information entryinterface 600 is shown. As shown, interface 600 includes images 601-Aand 601-B, which show the fixation apparatus and first and secondanatomical structure segments from different angles. In the example ofFIG. 6, image 601-A corresponds to an anteroposterior (AP) View, whileimage 601-B corresponds to a lateral (LAT) view. In some examples, thedisplayed images 601-A-B may be loaded and saved in computer memory, forexample in a library, database or other local collection of storedimages. The displayed images 601-A-B may then be selected and retrieved,acquired, and/or received from memory for display.

At operation 318, first image information is received, for example usingone or more graphical user interfaces of a computing system. The firstimage information may include indications of one or more locations,within the images, of at least part of one or more elements of thefixation apparatus. For example, the first image information may includeone or more indications of locations of struts, hinges, rings, and otherfixator elements. In some examples, the first image information may alsoinclude information about locations, within the images, of markerelements, for example that are mounted to components of the fixationapparatus, such as struts, hinges, and rings. In some cases, the firstimage information may include points representing locations of hingesand/or lines or vectors representing locations of struts. In someexamples, the first image information may be entered into a computingsystem by selecting or indicating one or more locations within thedisplayed images, for example using a mouse, keyboard, touchscreen orother user input devices. In particular, using one or more inputdevices, a user may select points or other locations in the images, drawlines, circles, and generate other graphical indications within theimages. For example, in some cases, a user may generate a point or smallcircle at a particular location in an image to indicate a location(e.g., center point) of a hinge within the image. As another example, insome cases, a user may generate a line and/or vector within an image toindicate a location and/or length of a strut within the image.

For example, as shown in FIG. 6, interface 600 includes six AP Viewstrut indicator buttons 611-A corresponding to each of the six struts ofthe fixation apparatus shown in AP View image 601-A. Each button 611-Aincludes text indicating a respective strut number (i.e., Strut 1, Strut2, Strut 3, Strut 4, Strut 5, Strut 6). Buttons 611-A may be selected bya user to indicate a strut for which first image information (e.g.,hinge locations, strut locations, etc.) will be provided by the user inAP View image 601-A. For example, in some cases, to provide first imageinformation for Strut 1 in AP View image 601-A, a user may first selectthe top strut indicator button 611-A (labeled with the text “Strut 1”)in order to indicate to the software that the user is about to providefirst image information for Strut 1 within AP View image 601-A. In somecases, the strut indicator button 611-A for Strut 1 may be pre-selectedautomatically for the user. Upon selection (or automatic pre-selection)of the strut indicator button 611-A for Strut 1, the user may proceed todraw (or otherwise indicate) a representation of Strut 1 within AP Viewimage 601-A. For example, in some cases, the user may use a mouse orother input device to select a location 621 (e.g., a center point) of aproximal hinge for Strut 1 within image 601-A. In some examples, theuser may then use a mouse or other input device to select a location 622(e.g., a center point) of the distal hinge of Strut 1 within image601-A. In some examples, the user may indicate the location and/orlength of Strut 1 by selecting the locations of the proximal and distalhinges and/or as the endpoints of a line or vector that represents thelocation and/or length of Strut 1. For example, as shown in FIG. 6, thesoftware may generate points or circles at the locations 621 and 622 ofthe proximal and distal hinges selected by the user within image 601-A.Additionally, the software may generate a line 623 representing thelocation and/or length of Strut 1 that connects the points or circles atthe locations 621 and 622 and of the proximal and distal hinges selectedby the user within image 601-A. Any other appropriate input techniquesmay also be employed by the user to indicate a location and/or length ofStrut 1 within image 610-A, such as generating line 623 by dragging anddropping a mouse, using a finger and/or pen on a touch screen, keyboard,and others. In some examples, the above described process may berepeated to draw points representing proximal and distal hinges andlines representing the locations and/or lengths of each of the sixstruts in the AP View image 601-A. Furthermore, the above describedprocess may also be repeated using LAT View strut indicator buttons611-B to draw points representing proximal and distal hinges and linesrepresenting the locations and/or lengths of each of the six struts inthe LAT View image 601-B.

In some examples, the first image information generated within images601-A and 601-B may include color-coded graphical representations of thestruts, for example to enable the graphical representations to be moreclearly associated with their respective struts. For example, in FIG. 6,the graphical representations (e.g., points, circles, and/or lines) ofStrut 1 in images 601A- and 601-B may be colored in red. This may matcha strut icon (which may also be colored red) displayed in the strutindicator buttons 611-A and 611-B for Strut 1 (displayed to the right ofthe text “Strut 1” in buttons 611-A and 611-B). As another example, inFIG. 6, the graphical representations (e.g., points, circles, and/orlines) of Strut 3 in images 601-A and 601-B may be colored in yellow.This may match a strut icon (which may also be colored yellow) displayedin the strut indicator buttons 611-A and 611-B for Strut 3 (displayed tothe right of the text “Strut 3” in buttons 611-A and 611-B).

FIG. 6 includes an AP View close-up assist checkbox 616-A and a LAT Viewclose-up assist checkbox 616-B, for example provided using one or moregraphical interfaces of a computing system. Selection of checkboxes616-A and 616-B may allow close-up views of areas of images 601-A and601-B surrounding the proximal and distal hinges of the struts that arecurrently being drawn by the user. This may enable more accurateindications of the locations (e.g., center points) of the hinges.Referring now to FIG. 7, close-up assist interface 700 depicts anotherAP View image 701 with the close-up assist being selected to provide aproximal hinge close-up assist view 702 and a distal hinge close-upassist view 703. As shown, proximal hinge close-up assist view 702provides an enlarged view of an area of AP View image 701 associatedwith the proximal hinge, while distal hinge close-up assist view 703provides an enlarged view of an area of AP View image 701 associatedwith the distal hinge. The user may manipulate (e.g., drag and drop) thelocation of the point/circle 721 in proximal hinge close-up assist view702 in order to more accurately depict the center point of the proximalhinge. The user may also manipulate (e.g., drag and drop) the locationof the point/circle 722 in distal hinge close-up assist view 703 inorder to more accurately depict the center point of the distal hinge. Asshould be appreciated, corresponding close-up assist views similar toviews 702 and 703 may also be provided for a respective LAT View image,for example using one or more graphical interfaces of a computingsystem.

Referring back to FIG. 6, to the right of buttons 611-A, are sixproximal hinge selector buttons 612-A. Additionally, to the right ofbuttons 612-A, are six distal hinge selector buttons 613-A. Furthermore,to the right of buttons 613-A, are six strut line selector buttons614-A. In some examples, buttons 612-A and/or 613-A may be selected touse the locations (e.g., center points) of the proximal and/or distalhinges indicated in AP View image 601-A in calculating positions andorientations of the first and the second anatomical structure segmentsand rings of the fixation apparatus in three-dimensional space (seeoperation 322). Additionally, in some examples, buttons 612-A and/or613-A may be selected to use the lines or vectors representing thelocation and/or length of struts indicated in AP View image 601-A incalculating positions and orientations of the first and the secondanatomical structure segments in three-dimensional space. Similarly,buttons 612-B, 613-B, and 614-B may be used to select the use oflocations (e.g., center points) of the proximal and/or distal hinges orstrut lines or vectors indicated in LAT View image 601-B in calculatingpositions and orientations of the first and the second anatomicalstructure segments in three-dimensional space.

Referring again to FIG. 3A, at operation 320, second image informationis received, for example using one or more graphical user interfaces ofa computing system. The second image information may include indicationsof one or more locations, within the images, of at least part of thefirst and the second anatomical structure segments. In some examples,the second image information may include indications of center lines ofthe first and the second anatomical structure segments and/or one ormore reference points (e.g., end points) of the first and the secondanatomical structure segments. In some examples, the second imageinformation may also include indications of locations of markerelements, for example implanted or otherwise associated with the firstand the second anatomical structure segments. In some examples, thesecond image information may be entered into a computing system byselecting or indicating one or more locations within the displayedimages, for example using a mouse, keyboard, touchscreen or other userinput devices. In particular, using one or more input devices, a usermay select points or other locations in the images, draw lines, circles,and generate other graphical indications within the images. For example,in some cases, a user may generate points or small circles at particularlocations in an image to indicate one or more reference points (e.g.,end points) of the first and the second anatomical structure segmentswithin the images. As another example, in some cases, a user maygenerate a line within an image to indicate a center line of the firstand the second anatomical structure segments within the images.

Referring now to FIG. 8A, an example second image information entryinterface 800 is shown. As shown, interface 800 includes AP View image601-A and LAT View image 601-B. Additionally, interface 800 includesbuttons 801-808, which may be used to assist in indication of anatomicalstructure center lines and reference points as will be described below.In particular, buttons 801 and 805 may be selected to indicate aproximal anatomical structure reference point in the AP View and LATView, respectively. Buttons 802 and 806 may be selected to indicate adistal anatomical structure reference point in the AP View and LAT View,respectively. Buttons 803 and 807 may be selected to indicate a proximalanatomical structure center line in the AP View and LAT View,respectively. Buttons 804 and 808 may be selected to indicate a distalanatomical structure center line in the AP View and LAT View,respectively. For example, as shown in FIG. 8A, a user may select button807 and then use one or more input devices to draw the center line 831for the proximal anatomical structure within LAT View image 601-B. Insome examples, the center line 831 may be colored red. Additionally, twoguidelines 832 are generated and displayed by the software on both sidesof the red center line. In some examples, the guidelines 832 may becolored green. These guidelines 832 may be displayed while the user isdrawing the center line 831 in order to assist the user in locating thecenter of the anatomical structure segment. The guidelines 832 may begenerated at equal distances from each side of the center line 831 andmay assist the user by, for example, potentially allowing the user tomatch (or nearly match) the guidelines 832 to sides of the anatomicalstructure segment. As shown in FIG. 8B, the user may select button 808and then use one or more input devices to draw the center line 841 forthe distal anatomical structure within LAT View image 601-B. As shown inFIG. 8C, the user may select button 803 and then use one or more inputdevices to draw the center line 851 for the proximal anatomicalstructure within AP View image 601-A. As shown in FIG. 8D, the user mayselect button 804 and then use one or more input devices to draw thecenter line 861 for the distal anatomical structure within AP View image601-A. As shown in FIGS. 8B-8D, guidelines 832 may also be displayed forassistance in drawing center lines 841, 851 and 861.

As shown in FIG. 8E, the user may select button 805 and then use one ormore input devices to indicate a reference point (e.g., end point) forthe proximal anatomical structure within LAT View image 601-B. As shown,a user has indicated a reference point 811 in LAT View image 601-B at anend point of the proximal anatomical structure segment. Additionally,upon indication of reference point 811, the software may generate anddisplay a corresponding dashed reference line 812 in AP View image601-A. The reference line 812 is a line drawn across AP View image 601-Athat passes through the location of the LAT View proximal referencepoint 811 within AP View image 601-A. The reference line 812 may,therefore, assist the user in determining the location of thecorresponding AP View proximal reference point, which may often be atthe intersection of the reference line 812 and the AP View proximalcenter line 851 within the AP View image 601-A. As shown in FIG. 8F, theuser may select button 801 and then use one or more input devices toindicate a reference point (e.g., end point) for the proximal anatomicalstructure within AP View image 601-A. In this example, the AP Viewproximal anatomical structure reference point 814 is indicated at theintersection of reference line 812 and the AP View proximal center line851 within the AP View image 601-A. The software may then generate anddisplay a corresponding dashed reference line 813 in the LAT View image601-B. The reference line 813 is a line drawn across LAT View image601-B that passes through the location of the AP View proximal referencepoint 814 within LAT View image 601-B. The reference line 813 may assistthe user by helping the user to confirm that the AP View reference point814 was placed correctly by showing how well it lines up relative to theLAT View reference point 811.

As shown in FIG. 8G, the user may select button 806 and then use one ormore input devices to indicate a reference point (e.g., end point) forthe distal anatomical structure within LAT View image 601-B. As shown, auser has indicated a reference point 815 in LAT View image 601-B at anend point of the distal anatomical structure segment. Additionally, uponindication of reference point 815, the software may generate and displaya corresponding dashed reference line 816 in AP View image 601-A. Thereference line 816 is a line drawn across AP View image 601-A thatpasses through the location of the LAT View distal reference point 815within AP View image 601-A. The reference line 816 may, therefore,assist the user in determining the location of the corresponding AP Viewdistal reference point, which may often be at the intersection of thereference line 816 and the AP View distal center line within the AP Viewimage 601-A. As shown in FIG. 8H, the user may select button 802 andthen use one or more input devices to indicate a reference point (e.g.,end point) for the distal anatomical structure within AP View image601-A. In this example, the AP View distal anatomical structurereference point 817 is indicated at the intersection of reference line816 and the AP View distal center line within the AP View image 601-A.The software may then generate and display a corresponding dashedreference line 818 in the LAT View image 601-B. The reference line 818is a line drawn across LAT View image 601-B that passes through thelocation of the AP View distal reference point 817 within LAT View image601-B. The reference line 818 may assist the user by helping the user toconfirm that the AP View reference point 817 was placed correctly byshowing how well it lines up relative to the LAT View reference point815.

Referring again to FIG. 3A, at operation 322, positions and orientationsof the first and second anatomical structure segments and rings of thefixation apparatus are determined in three-dimensional space. Forexample, in some cases, imaging scene parameters pertaining to fixator100, the anatomical structure segments 102, 104, imager(s) 130, andimage capturing devices 127, 129 are obtained. The imaging sceneparameters can be used in constructing a three-dimensionalrepresentation of the positioning of the anatomical structure segments102, 104 in the fixator 100, as described in more detail below. One ormore of the imaging scene parameters may be known. Imaging sceneparameters that are not known can be obtained, for example bymathematically comparing the locations of fixator elementrepresentations in the two-dimensional space of the x-ray images 126,128 to the three-dimensional locations of those elements on the geometryof the fixator 100. In a preferred embodiment, imaging scene parameterscan be calculated using a pin hole or perspective camera models. Forexample, the imaging scene parameters can be determined numericallyusing matrix algebra, as described in more detail below.

The imaging scene parameters can include, but are not limited to imagepixel scale factors, image pixel aspect ratio, the image sensor skewfactor, the image size, the focal length, the position and orientationof the imaging source, the position of the principle point (defined asthe point in the plane of a respective image 126, 128 that is closest tothe respective imager 130), positions and orientations of elements ofthe fixator 100, the position and orientation of a respective imagereceiver, and the position and orientation of the imaging source's lens.

In a preferred embodiment, at least some, such as all of the imagingscene parameters can be obtained by comparing the locations ofrepresentations of particular components, or fixator elements of thefixator 100 within the two-dimensional spaces of the images 126, 128,with the corresponding locations of those same fixator elements inactual, three-dimensional space. The fixator elements comprisecomponents of the orthopedic fixator 100, and preferably are componentsthat are easy to identify in the images 126, 128. Points, lines, conics,or the like, or any combination thereof can be used to describe therespective geometries of the fixator elements. For example, therepresentations of fixator elements used in the comparison could includecenter lines of one or more of the adjustable length struts 116, centerpoints of the universal joints 124, center points of the mountingmembers 114, and the like.

The fixator elements can further include marker elements that aredistinct from the above-described components of the fixator 100. Themarker elements can be used in the comparison, as a supplement to or inlieu of using components of the fixator 100. The marker elements can bemounted to specific locations of components of the fixator 100 prior toimaging, can be imbedded within components of the fixator 100, or anycombination thereof. The marker elements can be configured for enhancedviewability in the images 126, 128 when compared to the viewability ofthe other components of the fixator 100. For example, the markerelements may be constructed of a different material, such as aradio-opaque material, or may be constructed with geometries thatreadily distinguish them from other components of the fixator 100 in theimages 126, 128. In an example embodiment, the marker elements can havedesignated geometries that correspond to their respective locations onthe fixator 100.

Fixator elements can be identified for use in the comparison. Forexample, locations, within the images 126, 128 of fixator elements maybe indicated using the first image information received at operation 318and described in detail above. In some examples, the locations of thefixator elements in the two-dimensional space of the images 126, 128 maybe determined with respect to local origins 125 defined in the imagingplanes of the images 126, 128. The local origins 125 serve as a “zeropoints” for determining the locations of the fixator elements in theimages 126, 128. The locations of the fixator elements can be defined bytheir respective x and y coordinates with respect to a respective localorigin 125. The location of the local origin 125 within the respectiveimage can be arbitrary so long it is in the plane of the image.Typically, the origin is located at the center of the image or at acorner of the image, such as the lower left hand corner. It should beappreciated that the locations of the local origins are not limited toillustrated local origins 125, and that the local origins 125 can bealternatively defined at any other locations.

In some examples, a respective transformation matrix P may then becomputed for each of the images 126, 128. The transformation matricescan be utilized to map location coordinates of one or more respectivefixator elements in actual three-dimensional space to correspondinglocation coordinates of the fixator element(s) in the two-dimensionalspace of the respective image 126, 128. It should be appreciated thatthe same fixator element(s) need not be used in the comparisons of bothimages 126, 128. For example, a fixator element used in constructing thetransformation matrix associated with image 126 can be the same ordifferent from the fixator element used in constructing thetransformation matrix associated with image 128. It should further beappreciated that increasing the number of fixator elements used incomputing the transformation matrices can increase the accuracy method.The following equation represents this operation:

$\begin{matrix}{\begin{bmatrix}x \\y \\1\end{bmatrix} = {P \cdot \begin{bmatrix}X \\Y \\Z \\1\end{bmatrix}}} & (1)\end{matrix}$

The symbols x and y represent location coordinates, with respect to thelocal origin 125, of a fixator element point in the two-dimensionalspace of images 126, 128. The symbols X, Y and Z represent correspondinglocation coordinates, with respect to a space origin 135, of the fixatorelement point in actual three-dimensional space. In the illustratedembodiment, the point corresponding to the center of the plane definedby the upper surface of the upper fixator ring 106 has been designatedas the space origin 135. The illustrated matrix P can be at least fourelements wide and three elements tall. In a preferred embodiment, theelements of the matrix P can be computed by solving the following matrixequation:

A·p=B   (2)

The vectorp can contain eleven elements representing values of thematrix P. The following equations present arrangements of the elementsin the vector p and the matrix P:

$\begin{matrix}{p = \begin{bmatrix}p_{1} & p_{2} & p_{3} & p_{4} & p_{5} & p_{6} & p_{7} & p_{8} & p_{9} & p_{10} & p_{11}\end{bmatrix}^{T}} & (3) \\{P = \begin{bmatrix}p_{1} & p_{2} & p_{3} & p_{4} \\p_{5} & p_{6} & p_{7} & p_{8} \\p_{9} & p_{10} & p_{11} & p_{12}\end{bmatrix}} & (4)\end{matrix}$

In the preferred embodiment, the twelfth element p12 of the matrix P canbe set to a numerical value of one. The matrices A and B can beassembled using the two-dimensional and three-dimensional information ofthe fixator elements. For every point representing a respective fixatorelement, two rows of matrices A and B can be constructed. The followingequation presents the values of the two rows added to the matrices A andB for every point of a fixator element (e.g., a center point of arespective universal joint 124):

$\begin{matrix}{{\begin{bmatrix}\ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots \\X & Y & Z & 1 & 0 & 0 & 0 & 0 & {{- x} \cdot X} & {{- x} \cdot Y} & {{- x} \cdot Z} \\0 & 0 & 0 & 0 & X & Y & Z & 1 & {{- y} \cdot X} & {{- y} \cdot Y} & {{- y} \cdot Z} \\\ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots\end{bmatrix} \cdot p} = \begin{bmatrix}\ldots \\x \\y \\\ldots\end{bmatrix}} & (5)\end{matrix}$

The symbols X, Y and Z represent location coordinate values of a fixatorelement point in actual three-dimensional space relative to the spaceorigin 135, and the symbols x and y represent location coordinate valuesof the corresponding fixator element point in the two-dimensional spaceof the respective image 126, 128 relative to local origin 125.

For every line representing a respective fixator element, two rows ofmatrices A and B can be constructed. The following equation presents thevalues of the two rows added to the matrices A and B for every line of afixator element (e.g., a center line of a respective adjustable lengthstrut 116):

$\begin{matrix}{{\begin{bmatrix}\ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots \\{X \cdot a} & {Y \cdot a} & {Z \cdot a} & a & {X \cdot b} & {Y \cdot b} & {Z \cdot b} & b & {X \cdot c} & {Y \cdot c} & {Z \cdot c} \\{{dX} \cdot a} & {{dY} \cdot a} & {{dZ} \cdot a} & 0 & {{dX} \cdot b} & {{dY} \cdot b} & {{dZ} \cdot b} & 0 & {{dY} \cdot c} & {{dY} \cdot c} & {{dZ} \cdot c} \\\ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots\end{bmatrix} \cdot p} = \begin{bmatrix}\ldots \\{- c} \\0 \\\ldots\end{bmatrix}} & (6)\end{matrix}$

The symbols X, Y and Z represent location coordinate values of a pointbelonging to a line of a fixator element in actual three-dimensionalspace relative to the space origin 135. The symbols dX, dY and dZrepresent gradient values of the line in actual three-dimensional space.The symbols a, b and c represent constants defining a line in thetwo-dimensional space of a respective image 126, 128. For example, a, b,and c can be computed using two points belonging to a line on arespective image 126, 128. In a preferred embodiment, the value of b isassumed to be 1, unless the line is a vertical line, in which case thevalue of b is zero. A correlation of constants a, b and c with therespective image coordinates x and y is presented in the followingequation:

a·x+b·y+c=0   (7)

The equation (2) can be over constrained by using six or more fixatorelements, for example the adjustable length struts 116. It should beappreciated that it is not necessary for all of the fixator elements tobe visible in a single one of the images 126, 128 in order to obtain thematrix P. It should further be appreciated that if one or more of theabove-described imaging scene parameters are known, the known parameterscan be used to reduce the minimum number of the fixator elementsrequired to constrain equation (2). For instance, such information couldbe obtained from modern imaging systems in DICOM image headers.Preferably, a singular value decomposition or least squares method canbe used to solve equation (2) for values of the vector p.

In some examples, the transformation matrices may then be decomposedinto imaging scene parameters. The following equation can be used torelate the matrix P to matrices E and I:

P=I·E   (8)

It should be appreciated that additional terms can be introduced whendecomposing the matrix P. For example, the method presented by Tsai,described in “A Versatile Camera Calibration Technique for High-Accuracy3D Machine Vision Metrology Using of-the-shelf TV Cameras and Lenses”,IEEE Journal of Robotics & Automation, RA-3, No. 4, 323-344, August1987, which is incorporated herein by reference in its entirety, can beused to correct images 126, 128, for radial distortion.

Matrices E and I contain imaging scene parameters. The followingequation represents a composition of the matrix I:

$\begin{matrix}{I = \begin{bmatrix}{sx} & 0 & {- {tx}} \\0 & {sy} & {{- t}y} \\0 & 0 & {1/f}\end{bmatrix}} & (9)\end{matrix}$

The symbols sx and sy represent values of image coordinate scale factors(e.g., pixel scale factors). The symbol f, representing the focallength, corresponds to the value of the shortest distance between arespective imaging source 130 and the plane of a corresponding image126, 128. The symbols tx and ty represent the coordinates of theprinciple point relative to the local origin 125 of the respective image126, 128. The following equation represents the composition of thematrix E:

$\begin{matrix}{E = \begin{bmatrix}r_{1} & r_{2} & r_{3} & {- \left( {{r_{1} \cdot o_{x}} + {r_{2} \cdot o_{y}} + {r_{3} \cdot o_{z}}} \right)} \\r_{4} & r_{5} & r_{6} & {- \left( {{r_{4} \cdot o_{x}} + {r_{5} \cdot o_{y}} + {r_{6} \cdot o_{z}}} \right)} \\r_{7} & r_{8} & r_{9} & {- \left( {{r_{7} \cdot o_{x}} + {r_{8} \cdot o_{y}} + {r_{9} \cdot o_{z}}} \right)}\end{bmatrix}} & (10)\end{matrix}$

The symbols o_(x), o_(y) and o_(z) represent values of the position ofthe fixator 100 in actual three-dimensional space. The symbols r₁ to r₉describe the orientation of the fixator 100. These values can beassembled into a three-dimensional rotational matrix R represented bythe following equation:

$\begin{matrix}{R = \begin{bmatrix}r_{1} & r_{2} & r_{3} \\r_{4} & r_{5} & r_{6} \\r_{7} & r_{8} & r_{9}\end{bmatrix}} & (11)\end{matrix}$

The methods of Trucco and Verri, as described in “IntroductoryTechniques of 3-D Computer Vision”, Prentice Hall, 1998, or the methodof Hartley, as described in “Euclidian Reconstruction from UncalibratedViews”, Applications of Invariance in Computer Vision, pages 237-256,Springer Verlag, Berlin Heidelberg, 1994, which are incorporated hereinin their entireties, can be used to obtain values of the matrices Eand/or I. Utilizing the resulting values of matrices E and I, a completethree-dimensional imaging scene of the fixator 100 and the anatomicalstructure segments 102, 104 can be reconstructed.

For example, FIG. 2 illustrates an example three-dimensional imagingscene reconstructed from the x-ray images 126, 128. In the illustratedembodiment, x-rays are emitted from x-ray imagers 130. It should beappreciated that the x-ray imagers 130 can be the same or differentimagers, as described above. The x-rays emitted from the imagers 130 arereceived on by corresponding imaging devices, thus capturing the images126, 128. Preferably, the positioning of the imagers 130 with respect tothe local origins 125 is known.

In some examples, the images 126, 128 and the imaging scene parametersmay then be used to obtain the positions and/or orientations of theanatomical structure segments 102, 104 in three-dimensional space. Theposition and/or orientation data obtained can be used to develop atreatment plan for a patient, for example to change the orientationand/or position of the fractured first and second anatomical structuresegments 102, 104 in order to promote union between the anatomicalstructure segments 102, 104, as described in more detail below. Itshould be appreciated that the methods and techniques described hereinare not limited to applications of repositioning broken anatomicalstructures, and that orthopedic fixation with imagery analysis can beused in any other type of fixation procedure as desired, for examplelengthening of anatomical structures, correction of anatomical defects,and the like.

In some examples, anatomical structure elements comprisingrepresentations of particular portions (e.g., anatomical features) ofthe anatomical structure segments 102, 104, may then be identified andtheir locations within the images 126, 128 determined. For example,locations, within the images 126, 128 of the first and the secondanatomical structure segments may be indicated using the second imageinformation received at operation 320 and described in detail above. Insome examples, the locations of the anatomical structure elements may bedetermined with respect to the respective local origins 125 of images126, 128.

The anatomical structure elements can be used in the construction of thethree-dimensional representation of the position and/or orientation ofthe anatomical structure segments 102, 104. Preferably, the anatomicalstructure elements are easy to identify in the images 126, 128. Points,lines, conics, or the like, or any combination thereof can be used todescribe the respective geometries of the anatomical structure elements.For example, in the illustrated embodiment, points 134 and 136representing the fractured ends 103, 105 of the anatomical structuresegments 102, 104, respectively, are identified as anatomical structureelements in the images 126, 128.

The anatomical structure elements can further include marker elementsthat are implanted into the anatomical structure segments 102, 104 priorto imaging. The marker elements can be used as a supplement to or inlieu of the above-described anatomical structure elements identified inthe images 126, 128. The marker elements can be configured for enhancedviewability in the images 126, 128 when compared to the viewability ofanatomical features of the anatomical structure segments 102, 104. Forexample, the marker elements may be constructed of a radio-opaquematerial, or may be constructed with readily distinguishable geometries.

A three-dimensional representation 200 of the anatomical structuresegments 102, 104 can be reconstructed. The three-dimensionalrepresentation can be constructed with or without a correspondingrepresentation of the fixator 100. In the illustrated embodiment, pairsof ray-lines, such as ray lines 138, 140 and 142, 144 can be constructedfor the anatomical structure element points 134, 136, respectively. Eachray line connects an anatomical structure element in one of the images126, 128 with a respective imager 130. Each pair of ray lines can beanalyzed for a common intersection point, such as points 146, 148. Thecommon intersection points 146, 148 represent the respective positionsof the anatomical structure element points 134, 136, in thethree-dimensional representation of the anatomical structure segments102, 104. Of course more than a pair of ray lines, such as a plurality,can be constructed, for example if more than two images were captured.If the ray lines of a particular set do not intersect, a point closestto all the ray lines in the set can be used as the common intersectionpoint.

The positions and/or orientations of the anatomical structure segments102, 104 can be quantified or measured using common intersection points,for instance points 146, 148. For example, lines representing centerlines of the anatomical structure segments 102, 104 can be constructedand can be compared to the anatomical axes of the patient. Additionally,the distance between the fractured ends 103, 105 of the anatomicalstructure segments 102, 104 can be quantified. Using these or similartechniques, the positions and/or orientations of the anatomicalstructure segments 102, 104 can be determined. It is further noted that,in some examples, in addition to the positions and orientations of thefirst and second anatomical structure segments, the positions andorientation of rings (and/or other elements of the fixation apparatus)in three-dimensional space may also be determined, for example using anyof the techniques described. For example, in some cases, locations ofthe rings within the images 126, 128 may be determined based on thefirst image information and/or other provided information. In someexamples, these locations may then be used to determine the positionsand orientations of the rings in three-dimensional space. Additionally,in some examples, configuration information for the fixation apparatus,such as ring diameters and strut length and mounting information, mayalso be used to determine positions and orientations of the rings inthree-dimensional space.

Referring now to FIG. 3B, at operation 324, one or more deformityparameters are calculated. The deformity parameters may includeparameters relating to the deformity associated with the first andsecond anatomical structure segments. For example, in some cases, thedeformity parameters may include an amount of translation (e.g.,lateral, medial, anterior, and/or posterior), a degree of coronalangulation (e.g., valgus and/or varus), a degree of sagittal angulation,an amount by which anatomical structure length is too short and/or toolong, a degree of clinical rotational deformity (e.g., internal and/orexternal), and others. In some examples, the deformity parameters may becalculated as part of the process determining the positions andorientations of the first and segment anatomical structure segmentsdescribed above at operation 422, for example using the techniquesdescribed above with reference to operation 422.

At operation 326, the deformity parameters calculated at operation 424are displayed, for example using one or more graphical user interfacesof a computing system. Referring now to FIG. 9, a deformity parameterinterface 900 is shown. As shown, interface 900 includes various fields901-906 for displaying calculated values of various example deformityparameters, including AP View translation and coronal angulation, LATView translation and sagittal angulation, an amount by which anatomicalstructure length is too short or too long, and a degree of clinicalrotational deformity. In the example of FIG. 9, fields 901-905 each havea respective PFM badge 915 (including the text “PFM”) that is displayedto the left of each field 901-905. Each PFM badge 915 indicates that thevalue shown in the respective field 901-905 has been calculated by thesoftware. Interface 900 allows the deformity parameter values that aredisplayed in each field 901-906 to be edited by a user, for example bytyping a number in the fields 901-906 and/or by using number incrementcontrols 916 displayed to the right of each field 901-906. When a useredits a value that was calculated by the software, the PFM badge 915adjacent to the respective field may be removed to indicate that thevalue for the field has been edited by the user. In some examples, afterediting the values in one or more fields, the user may select RefreshPerspective Frame Matching Data button 920 to return each of the fieldsto the value that was calculated by the software. Also, in someexamples, after editing the values in one or more fields, the user mayselect Save and Update button 921 to cause the deformity parameters tobe recalculated based on the edited values provided by the user, forexample by repeating all or any portion of the calculations performed atoperation 322.

At operation 328, a graphical representation of the position andorientation of the first and the second anatomical structure segments isgenerated and displayed. The graphical representation of the positionand orientation of the first and the second anatomical structuresegments may be displayed using one or more graphical user interfaces ofa computing system. For example, as shown in FIG. 9, interface 900includes a graphical representation 950 of the position and orientationof the first and the second anatomical structure segments. Graphicalrepresentation 950 includes a representation 931 of the proximalanatomical structure segment and a representation 932 of the distalanatomical structure segment. In some examples, the graphicalrepresentation 950 may be generated based, at least in part, on thepositions and orientations of the first and segment anatomical structuresegments determined at operation 322. In some examples, when the useredits one or more deformity parameters and selects Save and Updatebutton 921, the graphical representation 950 may also be adjusted toreflect the saved edits to the deformity parameters. Graphicalrepresentation 950 may, for example, improve efficiency and reliabilityby providing the user with a visual confirmation of information enteredinto interface 900, for example to allow fast and easy identification oferrors or other problems.

At operation 330, one or more mounting parameters are calculated. Themounting parameters may include parameters relating to mounting of areference ring of the fixator onto a respective anatomical structuresegment. For example, in some cases, the mounting parameters may includean amount of offset (e.g., lateral, medial, anterior, and/or posterior)such as for a center of the reference ring with respect to a referencepoint, a degree of tilt (e.g., proximal and/or distal), an amount ofaxial offset, a master tab rotation, and others. In some examples, themounting parameters may be calculated as part of the process determiningthe positions and orientations of the first and segment anatomicalstructure segments described above at operation 322, for example usingthe techniques described above with reference to operation 322. It isnoted that, for the process of FIG. 3, the reference ring is notnecessarily required to be orthogonal with respect to the respectiveanatomical structure segment on which it is mounted. Thus, in someexamples, the reference ring may be non-orthogonal with respect to therespective anatomical structure segment on which it is mounted.

At operation 432, the mounting parameters calculated at operation 430are displayed, for example using one or more graphical user interfacesof a computing system. Referring now to FIG. 10, a mounting parameterinterface 1000 is shown. As shown, interface 1000 includes variousfields 1001-1006 for displaying calculated values of various examplemounting parameters, including AP View offset and tilt, LAT View offsetand tilt, axial offset, and master tab rotation. In the example of FIG.10, fields 1001-1006 each have a respective PFM badge 1015 that isdisplayed to the left of each field 1001-1006. Each PFM badge 1015indicates that the value shown in the respective field 1001-1006 hasbeen calculated by the software. Interface 1000 allows the mountingparameter values that are displayed in each field 1001-1006 to be editedby a user, for example by typing a number in the fields 1001-1006 and/orby using number increment controls 1016 displayed to the right of eachfield 1001-1006. When a user edits a value that was calculated by thesoftware, the PFM badge 1015 adjacent to the respective field may beremoved to indicate that the value for the field has been edited by theuser. In some examples, after editing the values in one or more fields,the user may select Refresh Perspective Frame Matching Data button 1020to return each of the fields to the value that was calculated by thesoftware. Also, in some examples, after editing the values in one ormore fields, the user may select Save and Update button 1021 to causethe deformity parameters to be recalculated based on the edited valuesprovided by the user, for example by repeating all or any portion of thecalculations performed at operation 322.

At operation 334, a graphical representation of the position andorientation of the reference ring and the respective anatomicalstructure segment to which it is mounted is generated and displayed. Thegraphical representation of the position and orientation of thereference ring and the respective anatomical structure segment may bedisplayed using one or more graphical user interfaces of a computingsystem. For example, as shown in FIG. 10, interface 1000 includes agraphical representation 1050 of the position and orientation of thereference ring and the respective anatomical structure segment.Graphical representation 1050 includes a representation 1031 of theproximal anatomical structure segment, a representation 1033 of theproximal (reference) ring, and a representation 1032 of the distalanatomical structure segment. In some examples, the graphicalrepresentation 1050 may be generated based, at least in part, on thepositions and orientations of the reference ring and the respectiveanatomical structure segment determined at operation 322. The graphicalrepresentation of the reference ring and the respective anatomicalstructure segment may, therefore, reflect and/or indicate the positionsand orientations of reference ring and the respective anatomicalstructure segment determined at operation 322. In some examples, whenthe user edits one or more mounting parameters and selects Save andUpdate button 1021, the graphical representation 1050 may also beadjusted to reflect the saved edits to the mounting parameters.Graphical representation 1050 may, for example, improve efficiency andreliability by providing the user with a visual confirmation ofinformation entered into interface 1000, for example to allow fast andeasy identification of errors or other problems.

At operation 336, one or more treatment plan options are received, forexample using one or more graphical user interfaces of a computingsystem. A treatment plan is a plan for manipulating the fixationapparatus, for example in order to correct the deformity of the firstand the second anatomical structure segments. The treatment plan mayinclude, for example, a plan for making gradual adjustments to thepositions and orientations of the fixator rings with respect to eachother, for example by changing the lengths of the struts of the fixationapparatus. Referring now to FIG. 11, an example treatment plan interface1100A is shown. The interface 1100A includes controls for selecting, bya user, various treatment plan options. In particular, controls 1101and/or 1102 allow selecting of a treatment plan start date, control 1103allows selection of an option to perform axial movement first (e.g., inan initial part of the treatment, such as prior to rotational movement),control 1104 allows selection of an option to indicate a final distancebetween reference points, control 1105 allows selection of an option tocalculate the treatment plan based on a specified duration (e.g., anumber of days) for axial movement, control 1106 allows selection of anoption to calculate the treatment plan based on a rate of distraction atthe reference point (e.g., for example millimeters (mm)/day) for axialmovement, control 1108 allows selection of an option to calculate thetreatment plan based on a specified duration (e.g., a number of days)for deformity correction, control 1109 allows selection of an option tocalculate the treatment plan based on a rate of distraction at thereference point (e.g., for example millimeters (mm)/day) for deformitycorrection, and control 1107 allows selection of an option to performtwo adjustments per day. In some examples, when control 1007 is notselected, a default option of one adjustment per day may be used. Insome examples, after selecting desired treatment plan options, the usermay select Update Adjustment Plan button 1110 to trigger generation ofthe treatment plan. Additionally, after initial generation of thetreatment plan, the user may also be permitted to adjust the treatmentplan options and have the treatment plan re-generated with the adjustedoptions by re-selecting Update Adjustment Plan button 1110

At operation 338, manipulations to the fixation apparatus for correctionof the anatomical structure deformity (i.e., a treatment plan) aredetermined. The manipulations to the fixation apparatus may includeadjustments to the struts of the fixation apparatus, such as adjustmentsto the sizes and/or lengths of the struts. In some examples, operation338 may be performed based, at least in part, on the treatment planoptions received at operation 336. For example, operation 338 may beperformed based, at least in part, on specified start date, oninstructions to perform axial movement first (e.g., in an initial partof the treatment, such as prior to rotational movement), a specifiedfinal distance between reference points, instructions to performadditional lengthening by a specified amount, instructions to generatean axial gap to ensure anatomical structure clearance, a specifiedduration (e.g., a number of days) of treatment, a specified rate ofdistraction, and/or instructions to perform two perform a specifiedquantity (e.g., one, two, etc.) of adjustments per day.

In some examples, the treatment plan may also be determined based, atleast in part, on a determination of desired changes to the positionsand/or orientations of the anatomical structure segments 102, 104, forinstance how the anatomical structure segments 102, 104 can berepositioned with respect to each other in order to promote unionbetween the anatomical structure segments 102, 104. For example, in somecases, it may be desirable to change the angulation of the secondanatomical structure segment 104 such that the axes L1 and L2 arebrought into alignment, and to change the position of the secondanatomical structure segment such that the fractured ends 103, 105 ofthe anatomical structure segments 102, 104 abut each other. Once thedesired changes to the positions and/or orientations of the anatomicalstructure segments 102, 104 have been determined, a treatment plan foreffecting the position and/or orientation changes can be determined. Ina preferred embodiment, the desired changes to the positions and/ororientations of the anatomical structure segments 102, 104 can beeffected gradually, in a series of smaller changes. The positions and/ororientations of the anatomical structure segments 102, 104 can bechanged by changing the positions and/or orientations of the upper andlower fixator rings 106, 108 with respect to each other, for instance bylengthening or shortening one or more of the length adjustable struts116.

The required changes to the geometry of the fixator 100 (i.e., theposition and/or orientation of the fixator 100) that can enable thedesired changes to the positions and/or orientations of the anatomicalstructure segments 102, 104 can be computed using the matrix algebradescribed above. For example, the required repositioning and/orreorientation of the second anatomical structure segment 104 withrespect to the first anatomical structure segment 102 can be translatedto changes in the position and/or orientation of the lower fixator ring108 with respect to the upper fixator ring 106.

At operation 340, indications of the determined manipulations to thefixation apparatus are provided to one or more users. For example, insome cases, indications of the determined manipulations to the fixationapparatus may be provided using one or more graphical user interfaces ofa computing system, using a printed hard copy, using audio feedback,and/or using other techniques. In particular, referring now to FIG. 12,it is seen that indications of the determined manipulations to thefixation apparatus may be provided within interface 1100B. Specifically,selection of Strut Adjustment Plan tab 1122 may cause treatment planinterface 1100B to provide a chart 1130, including day-by-daymanipulation information for each strut within the fixation apparatus.In this example, chart 1130 shows a length for each strut on each day oftreatment. In some examples, one or more alerts may be generated for oneor more manipulations to the fixation apparatus that result in at leastone of strut movement of more than a threshold amount. For example, insome cases, strut movements exceeding particular threshold amount (e.g.,3 mm per day), which may be referred to as rapid strut movements, may beindicated by displaying a red triangle icon next to the indication ofthe strut movement in chart 1130. As also shown in FIG. 12, a PDFversion of the chart 1130 may be generated by selecting View Draft PDFbutton 1131. The generated PDF may, in some examples, be printed tocreate a hard copy version of chart 1130.

In the example of FIG. 12, chart 1130 includes blocks 1132-A and 1132-Bindicating ranges of dates on which changes of strut sizes, referred toas strut swaps, may be performed. In particular, block 1132-A indicatesthat a strut swap may be performed for Strut 4 on Day 0 through Day 2,while block 1132-B indicates that a strut swap may be performed forStrut 4 on Day 3 through Day 14 (and subsequent days). In some examples,blocks 1132-A and 1132-B may be color-coded to match a color assigned toa respective strut. For example, blocks 1132-A and 1132-B may be coloredgreen to match a green color that may be assigned to Strut 4. Referringnow to FIG. 13, Strut Swaps Calendar tab 1123 of treatment planinterface 1100-C may be selected to generate a calendar 1140 indicatingranges of dates on which strut swaps may be performed.

In some examples, the struts of the fixation apparatus attached to thepatient may be color-coded, for example using color-coded caps, marker,or other color-coded materials included within and/or attached to thestruts. In some examples, the physical color-coding of the struts in thefixation apparatus attached to the patient may match the color-coding ofstruts used in the software. For example, the physical color-coding ofthe struts in the fixation apparatus may match the color-coding ofstruts that may be used to color-code the blocks 1132-A and 1132-B ofchart 1130, graphical representation 520, and other color-codedrepresentations of the struts displayed by the software. In someexamples, this may make it easier for physicians and/or patients toconfirm that, when they physically adjust a strut on the fixationapparatus, they are adjusting the correct strut by the correct amount.

At operation 342, one or more graphical representations of the positionand orientation of the first and second anatomical structure segmentsand the rings of the fixation apparatus is generated and displayed. Thegraphical representation of the position and orientation of the firstand the second anatomical structure segments and the rings of thefixation apparatus may be displayed using one or more graphical userinterfaces of a computing system. For example, referring back to FIG.11, selection of Treatment Simulation tab 1121 may cause interface 1100to display a graphical representation 1150 of the position andorientation of the first and the second anatomical structure segmentsand the rings of the fixation apparatus. Graphical representation 1150includes a representation 1031 of the proximal anatomical structuresegment, a representation 1033 of the proximal (reference) ring, arepresentation 1032 of the distal anatomical structure segment, and arepresentation 1034 of the distal ring. In some examples, the one ormore graphical representations of the position and orientation of thefirst and second anatomical structure segments and the rings of thefixation apparatus may include day-by-day graphical representations ofthe position and orientation of the first and second anatomicalstructure segments and the rings of the fixation apparatus throughouttreatment for the anatomical structure deformity. For example, as shownin FIG. 11, a user may select a particular day of treatment for which togenerate and display a graphical representation 1150 using controls1151, 1152, 1153, and/or 1154. For example, control 1151 may be selectedto allow incrementing of the selected day, control 1154 may be selectedto allow decrementing of the selected day, and slider 1152 may be slidalong bar 1153 to increment and/or decrement the selected day. It isalso noted that slider 1152 displays an indication of the currentlyselected day, which, in the example of FIG. 11, is treatment day zero.Thus, in FIG. 11, graphical representation 1150 shows the position andorientation of the first and second anatomical structure segments andthe rings of the fixation apparatus at treatment day zero. Usingcontrols 1151-1154 to select a different day of treatment may causegraphical representation 1150 to be adjusted to show the position andorientation of the first and second anatomical structure segments andthe rings of the fixation apparatus on the selected different day. Asshould be appreciated, allowing the surgeon and/or patient to seegraphical representations of the position and orientation of the firstand second anatomical structure segments and the rings of the fixationapparatus throughout treatment may be beneficial by, for example,providing an additional visual tool to improve accuracy and assist inplanning of treatment. Additionally, graphical representation 1150 (aswell as graphical representations described herein) may, for example,improve efficiency and reliability by providing the user with a visualconfirmation of information entered into interface 1100, for example toallow fast and easy identification of errors or other problems. It isfurther noted that the view of graphical representation 1150 (as well asother graphical representations described herein) may be rotated (forexample by a complete 360 degrees), zoomed in and out, moved indirection, and otherwise manipulated, for example using controls1181-1184 adjacent to the upper right side of the graphicalrepresentation 1150. This may allow views of the first and secondanatomical structure segments and/or the rings of the fixation apparatusfrom various orientations that may not be available, or may be difficultto obtain, using x-rays and other imaging techniques, thereby alsoimproving reliability and accuracy and providing additional visualconfirmation of calculated values. In particular, view of the graphicalrepresentation 1150 may be rotated using control 1181, zoomed in usingcontrol 1182, zoomed out using control 1183, and panned using control1184. Also, in some examples, other controls, such as a mouse andtouchscreen, may also be employed to rotate, zoom, pan, and otherwisemanipulate graphical representation 1150. Additionally, in someexamples, control 1185 may be used to select an anteroposterior (AP)view, control 1186 may be used to select a lateral view, and control1187 may be used to select a proximal view.

At operation 344, the treatment plan may be implemented, that is thegeometry of the fixation apparatus may be changed, for example based onthe manipulations determined at operation 338, in order to changepositions and orientations of the anatomical structure segments.

Guided Frame Matching with Graphical Fixator Projection

As described above, a frame matching process may be employed todetermine positions and orientations of anatomical structure segments inthree-dimensional space, such as for generating a treatment plan forcorrection of an anatomical deformity. As also described above, in someexamples, as part of the frame matching process, a surgeon or other usermay identify locations of fixator elements (e.g., rings, struts, etc.)within displayed images (e.g., x-rays) that show the fixator attached tothe anatomical structure segments. Some examples of this process aredescribed above with reference to operation 318 of FIG. 3A and FIG. 6.For example, as shown in FIG. 6 and described above, a user may identifylocations of struts within AP View image 601-A and LAT View image 601-Bas part of the frame matching process. However, it may often bedifficult for the user to identify and mark positions of certain fixatorelements, such as struts, within the images. In particular, dependingupon the location and orientation from which an image is captured,struts and other fixator elements may be not be identified easily, suchas because they may wholly or partially overlap one another or mayotherwise be obscured within the images. For example, in some cases, itmay often be more difficult to identify struts within a lateral imagethan to identify the struts within an anterior image, such as becausethe struts may often overlap one another when viewed within the lateralimage. For example, as shown in FIG. 6, several of the struts arepositioned within LAT View image 601-B such that they are displayed veryclose together. Specifically, Strut 1 and Strut 2 are displayed veryclose together on the left side of the image 601-B, Strut 3 and Strut 6are displayed very close together in the middle side of the image 601-B,and Strut 4 and Strut 5 are displayed very close together on the rightside of the image 601-B. This may make it cumbersome for the user toidentify, for example, Strut 1 as opposed to Strut 2, Strut 3 as opposedto Strut 6, or Strut 4 as opposed to Strut 5.

In some examples, to alleviate the above and other problems, a guidedframe matching technique may be employed. Some examples of the guidedframe matching technique will now be described with reference to FIGS.14-17. Specifically, referring now to FIG. 14, an example process forproviding a graphical projection of a fixator using a guided framematching technique will now be described in detail. As described above,the fixator may include fixator elements such as rings and struts andmay be for correcting a deformity of first and second anatomicalstructure segments. The process of FIG. 14 is initiated at operation1410, at which first and second images of the first and the secondanatomical structure segments and the fixator attached thereto aredisplayed. The first and the second images may have respective imageplanes. As shown in FIG. 2 and described above, there is an angle abetween the image planes of the images 126, 128.

A first example of the display of the first and the second images atoperation 1410 is shown in FIG. 6, which includes AP View image 601-Aand LAT View image 601-B as described above. An additional example ofthe display of the first and the second images at operation 1410 isshown in FIG. 15A, which will now be described in detail. In particular,FIG. 15A displays an AP View image 1501-A and a LAT View image 1501-B,which are images of a fixator 1510 including proximal fixator ring 1511,distal fixator ring 1512 and fixator struts 1513. The images 1501-A and1501-B show the fixator 1510 attached to a first anatomical structuresegment 1521 and a second anatomical structure segment 1522. The firstand second images of the first and the second anatomical structuresegments and the fixator attached thereto may be displayed at operation1410 using one or more graphical user interfaces of a computing system.For example, images 1501-A and 1501-B of FIGS. 15A-17B may be displayedusing one or more graphical user interfaces of a computing system. It isnoted that any, or all, of the contents shown in each of FIGS. 15A-17Bmay be displayed using one or more graphical user interfaces of acomputing system.

It is noted that, in the examples of FIGS. 15A-17B, the images 1501-Aand 1501-B are simulated images-as opposed to actual x-rays (as in FIG.6) or other images captured from an imager or imaging source. It isnoted that the simulated images of FIGS. 15-17 are provided merely forease of illustration of the concepts described herein. In practice, theimages 1501-A and 1501-B may be non-simulated images, such as x-rays,which are captured using an imager, imaging source, x-ray imager, cameraor other image capture device, and that show an actual fixator that isphysically attached to an actual anatomical structure segment (such asshown in FIG. 6). Thus, even though images 1501-A and 1501-B aredisplayed as simulations, the concepts described herein should beunderstood to also be applicable to non-simulated images (i.e., imagesthat were captured using an imager, imaging source, x-ray imager, cameraor other image capture device) similar to the images 601-A and 601-B ofFIG. 6.

At operation 1412, first indications are received of first locations,within the first image, of a plurality of elements of the fixator, suchas struts of the fixator, for example using the one or more graphicaluser interfaces of the computing system. For example, as described abovewith respect to FIG. 6, the user may indicate locations of struts withinthe AP View image 601-A, such as by clicking on endpoints of the struts(e.g., hinges) using an attached mouse or other input device. Asdescribed above, the strut indicator button 611-A for Strut 1 may bepre-selected automatically for the user. Upon selection (or automaticpre-selection) of the strut indicator button 611-A for Strut 1, the usermay proceed to draw (or otherwise indicate) a representation of Strut 1within AP View image 601-A. For example, in some cases, the user may usea mouse or other input device to select a location 621 (e.g., a centerpoint) of a proximal hinge for Strut 1 within image 601-A. In someexamples, the user may then use a mouse or other input device to selecta location 622 (e.g., a center point) of the distal hinge of Strut 1within image 601-A. In some examples, the user may indicate the locationand/or length of Strut 1 by selecting the locations of the proximal anddistal hinges and/or as the endpoints of a line or vector thatrepresents the location and/or length of Strut 1. For example, as shownin FIG. 6, the software may generate points or circles at the locations621 and 622 of the proximal and distal hinges selected by the userwithin image 601-A. Additionally, the software may generate a line 623representing the location and/or length of Strut 1 that connects thepoints or circles at the locations 621 and 622 and of the proximal anddistal hinges selected by the user within image 601-A. Any otherappropriate input techniques may also be employed by the user toindicate a location and/or length of Strut 1 within image 610-A, such asgenerating line 623 by dragging and dropping a mouse, using a fingerand/or pen on a touch screen, keyboard, and others. In some examples,the above described process may be repeated to draw points representingproximal and distal hinges and lines representing the locations and/orlengths of each of the six struts in the AP View image 601-A. A similartechnique may also be employed to indicate the locations of each of thesix fixator struts 1513 in AP View image 1501-A of FIG. 15A.

In some examples, after the user indicates locations of the struts 1513within the AP View image 1501-A, the software may use the indicatedstrut locations to determine locations of the fixator rings 1511 and1512 within the AP View image 1501-A. The software may then generatering graphical representations 1531 and 1532, corresponding to thefixator rings 1511 and 1512, respectively, and display the ringgraphical representations 1531 and 1532 at the determined locations ofthe fixator rings 1511 and 1512 within the AP View image 1501-A.Referring now to FIG. 15B, it is seen that ring graphicalrepresentations 1531 and 1532 are generated by the software anddisplayed within AP View image 1501-A at the corresponding locations ofthe respective fixator rings 1511 and 1512. It is noted that the fixatorring graphical representations 1531 and 1532 are shown in FIG. 15B witha different shade/color than the actual fixator rings 1511 and 1512 toindicate that the fixator ring graphical representations 1531 and 1532are generated by the software and are not included in the actualunderlying AP View image 1501-A. Specifically, the fixator ringgraphical representations 1531 and 1532 are shown in blue color/shades,while the fixator rings 1511 and 1512 are shown in black color/shades.The presence of the fixator ring graphical representations 1531 and 1532in the AP View image 1501-A of FIG. 15B (but not in the LAT View image1501-B of FIG. 15B) indicates that fixator element location informationhas been received for the AP View image 1501-A of FIG. 15B but has notyet been received for the LAT View image 1501-B of FIG. 15B.

At operation 1414, a graphical projection of the fixator is overlaid,for example using the one or more graphical user interfaces of thecomputing system, on the second image. The graphical projection of thefixator may include, for example, graphical representations of the ringsand/or the struts of the fixator. For example, referring now to FIG. 16,it is seen that a graphical projection 1600 of the fixator is displayedthat includes a graphical representation 1611 of the proximal ring and agraphical representation 1612 of the distal ring. As shown, thegraphical projection 1600, including graphical representations 1611 and1612 is overlaid on the second image, which in this example is the LATView image 1501-B. Although not shown in FIG. 16, in addition or as analternative to graphical representations 1611 and 1612 of the fixatorrings, the graphical projection 1600 may also include graphicalrepresentations of other fixator elements, such as fixator struts.Moreover, in some examples, the graphical representations of the fixatorstruts included in a graphical projection may be color coded such thatdifferent struts are shown in different colors, for example to matchdifferent colors of strut indicator buttons 611-A and 611-B.

The graphical projection 1600 of the fixator may be rotated relative tothe first locations of the plurality of fixator elements identified inthe first image. Specifically, the graphical projection 1600 of thefixator may be rotated relative to the first locations based at least inpart on an angle (such as at the exact angle or at an approximation ofthe angle) of image planes of the first and the second images withrespect to one another. As shown in FIG. 2 and described above, there isan angle a between the image planes of the images 126, 128. Thus, in theexample of FIG. 16, the AP View image 1501-A may have a respective APView image plane, and the LAT View image 1501-B may have a respectiveLAT View image plane at an angle of ninety degrees with respect to theAP View image plane. Accordingly, in the example of FIG. 16, thegraphical projection 1600 of the fixator is rotated ninety degreesrelative to the first locations of the plurality of fixator elementsidentified in the first image. For example, both the proximal ringgraphical representation 1611 and the distal ring graphicalrepresentation 1612 of FIG. 16 are rotated ninety degrees relative tothe proximal fixator ring 1511 (and/or the respective ringrepresentation 1531) and distal fixator ring 1512 (and/or the respectivering representation 1532) in the AP View image 1501-A.

The graphical projection 1600 of the fixator may be rotated based atleast in part on the angle between image planes of the images becausethat rotation may correspond to the expected position of the fixator inthe second image. For example, if an image plane of the LAT View image1501-B is at an angle of ninety degrees to an image plane of the AP Viewimage 1501-A, then it may be expected that the locations of the fixatorrings in the LAT View image 1501-B will be rotated ninety degreesrelative to the locations of the fixator rings in the AP View image1501-A. In this way, the overlaying of the graphical projection 1600 onthe second image may assist the user in identifying locations of theplurality of fixator elements in the second image. In some examples, auser may provide a numerical value, such as a quantity of degrees (e.g.,ninety degrees), that expressly indicates to the software the value ofthe angle between image planes of the images. In other examples, thevalue of the angle may be inferred by the software based on descriptionsof the images (e.g., anteroposterior, anterior, posterior, lateral,medial, etc.) or using other techniques. In the examples of FIGS. 15-17,image 1501-A is an AP View image and image 1501-B is a lateral image. Itis noted, however, that the guided frame matching techniques describedherein may be used between any different combinations of images takenfrom any directions and orientations and having image planes at anyangle with respect to one another.

Additionally, it is noted that the software may also manipulate otherfeatures of the graphical projection 1600 (e.g. size, location,orientations, etc.) such as to correct for other differences (e.g.,location, orientation, zoom level, etc.) between the first and thesecond images. For example, in some cases, if the second image wascaptured from a closer location to the fixator and/or is more zoomed-inthan the first image, then the software may correct for this byenlarging the size of the graphical projection 1600 relative to the sizeof the fixator elements in the first image. By contrast, in some cases,if the second image was captured from a further location from thefixator and/or is more zoomed-out than the first image, then thesoftware may correct for this by reducing the size of the graphicalprojection 1600 relative to the size of the fixator elements in thefirst image.

Thus, in some examples, the graphical projection 1600 of the fixator maybe generated based, at least in part, on the first locations of theplurality of fixator elements in the first image indicated at operation1412. Additionally or alternatively, in some examples, the graphicalprojection 1600 of the fixator may be generated based, at least in part,on configuration information for the fixator that is provided to thesoftware by the user, such as ring types (e.g., full ring, foot plate,etc.), ring sizes, strut lengths, indications of mounting points (e.g.,ring holes), and other information. Various types of configurationinformation and techniques for providing such information to thesoftware are described in detail above, such as with respect to FIG. 5and operation 314 of FIG. 3A, and are not repeated here.

At operation 1416, the software may allow a user to manipulate (e.g.,resize, rotate, move, etc.) the graphical projection and/or the secondimage. For example, the user may manipulate the graphical projection tomake it more precisely align with the positions of the fixator elementsin the second image. For example, the software may provide controls thatallow resizing (making the graphical projection larger or smaller) orrotating of the graphical projection relative to its initial placementby the software when being overlaid upon the second image at operation1414. For example, in some cases, it may be necessary to resize and/orrotate the graphical projection to correct for slight differences in theactual angle between the first and the second images relative to theexpected angle (e.g., if the images are actually at an angle ofninety-two degrees rather than ninety degrees, etc.), to correct fordifferences in distance, position or orientation of the first and thesecond images relative to the objects included in the images, or forother reasons. In some examples, the software may provide variouscontrols, such as buttons, that allow selections of operations such asmove, resize and rotate, and the software may be configured to receiveinput from input devices, such as a mouse or keyboard, to accomplishthose manipulations, for example via drag-and-drop, button clicks,keystrokes, etc.

In some examples, in addition or as an alternative to allowing a user tomanipulate the graphical projection, the software may allow the user tomanipulate the second image (e.g., LAT View image 1501-B) upon which thegraphical projection is overlaid. For example, in some cases, thesoftware may allow the user to resize, rotate and/or move the secondimage and/or elements shown within the second image, such as to assistin aligning the fixator elements shown in the second image withcorresponding elements of the graphical projection. Referring now toFIG. 17A, it is seen that the user has manipulated the second image,which is LAT View image 1501-B, by moving the LAT View image 1501-B downand to the right from its prior screen/interface location shown in FIG.16. By moving the LAT View image 1501-B in this manner (without movingthe graphical projection 1600), this allows the fixator elements in theLAT View image1501-B to be moved down and to the right such that theyalign with corresponding elements of graphical projection 1600. Forexample, as shown in FIG. 17A, the graphical representations 1611 and1612 of the fixator rings substantially align with the respectivefixator rings 1511 and 1512. Thus, only small portions of the fixatorrings 1511 and 1512 are visible in FIG. 17 because they have been almostentirely overlaid by the respective graphical representations 1611 and1612 of the fixator rings. In particular, in FIG. 17A, proximal ringgraphical representation 1611 substantially aligns with (and almostentirely overlays) proximal fixator ring 1511, and distal ring graphicalrepresentation 1612 substantially aligns with (and almost entirelyoverlays) distal fixator ring 1512.

At operation 1418, second indications are received of second locations,within the second image, of the plurality of elements of the fixator,for example using the one or more graphical user interfaces of thecomputing system. For example, in some cases, once the graphicalprojection 1600 of the fixator has been satisfactorily aligned with theposition of the fixator in the second image (e.g., in the LAT View image1501-B), the user may employ the graphical projection 1600 as a guide toassist in identifying the locations of the plurality of fixator elementsin the second image (e.g., in the LAT View image 1501-B). For example,although not shown in FIGS. 16 and 17A, the graphical projection 1600 ofthe fixator may include graphical representations of the fixator struts.In these examples, when the user aligns the graphical projection of thefixator with the position of the fixator in the second image, thegraphical representations of the fixator struts in the graphicalprojection will align with (and overlay) the respective locations of thefixator struts within the second image. As also noted above, graphicalrepresentations of the fixator struts in the graphical projection may becolor coded such that different struts are shown in different colors,for example to match different colors of strut indicator buttons 611-Aand 611-B and different colors of struts in the first image. The colorcoding of the graphical representations of the struts in the graphicalprojection may therefore assist the user in distinguishing betweenrespective struts that are positioned closely together in the secondimage. For example, as shown in FIG. 17A, a first strut 1513-A and asecond strut 1513-B are positioned closely together on the right side ofthe LAT View image 1501-B. If graphical representations of these strutsare included in the graphical projection 1600 and are aligned with (andoverlaying) the struts 1513-A and 1513-B, then this may assist the userin distinguishing between the struts 1513-A and 1513-B in the LAT Viewimage, particularly if the graphical representations of the struts inthe graphical projection 1600 are color coded. For example, if agraphical representation of the first strut 1513-A is shown in red, thenthe user may immediately appreciate that the red colored strutrepresentation is aligned with and overlaying the first strut 1513-A.Additionally, if a graphical representation of the second strut 1513-Bis shown in orange, then the user may immediately appreciate that theorange colored strut representation is aligned with and overlaying thesecond strut 1513-B. In the manner, the user may employ graphicalrepresentations of the struts (and/or other fixator elements) in thegraphical projection as a guide to identify respective ones of theplurality of fixator elements in the second image. It is noted that, inaddition to the struts, graphical representations 1611 and 1612 of therings may also be color coded.

Accordingly, by using the graphical projection 1600 of the fixator as aguide, the user may identify locations of the plurality of the fixatorelements in the second image. The user may then indicate the locationsof these fixator elements to the software, such as by using the same ora similar process as was used to identify the plurality of fixatorelements in the first image at operation 1412. For example, thetechniques described above in operation 1412 (e.g., identifying proximaland distal hinges or endpoints of each strut) for the first image (APView images 601-A and 1501-A) may be repeated for the second image (LATView images 601-B and 1501-B), such as by using LAT View strut indicatorbuttons 611-B to draw points representing proximal and distal hinges andlines representing the locations and/or lengths of each of the sixstruts in the LAT View images 601-B and 1501-B.

In some examples, after the user has indicated locations of the struts1513 and/or other fixator elements within the LAT View image 1501-A atoperation 1418 (e.g., using the graphical projection 1600 of the fixatoras a guide), the software may use the indicated fixator elementlocations to determine locations of the fixator rings 1511 and 1512within the LAT View image 1501-B. The software may then generate ringgraphical representations 1731 and 1732, corresponding to the fixatorrings 1511 and 1512, respectively, and display the ring graphicalrepresentations 1731 and 1732 at the determined locations of the fixatorrings 1511 and 1512 within the LAT View image 1501-B. Referring now toFIG. 17B, it is seen that ring graphical representations 1731 and 1732are generated by the software and displayed within LAT View image 1501-Bat the corresponding locations of the respective fixator rings 1511 and1512.

At operation 1420, the first locations of the plurality of fixatorelements in the first image (indicated at operation 1412) and the secondlocations of the plurality of fixator elements in the second image(indicated at operation 1418 using the graphical projection as a guide)are used to determine positions and orientations of the first and secondanatomical structure segments in three-dimensional space. For example,as described in detail above with respect to operation 322 of FIG. 3A,imaging scene parameters may be used to determine positions andorientations of the first and second anatomical structure segments inthree-dimensional space. As also described above, the imaging sceneparameters may be obtained by comparing the locations of representationsof particular components, or fixator elements of the fixator within thetwo-dimensional spaces of the first and the second images, with thecorresponding locations of those same fixator elements in actual,three-dimensional space. As also described above, such as with respectto operation 338 of FIG. 3B, manipulations to the fixation apparatus forcorrection of the anatomical structure deformity (i.e., a treatmentplan) may be determined using the positions and orientations of thefirst and second anatomical structure segments in three-dimensionalspace. Specifically, the treatment plan may be determined based, atleast in part, on a determination of desired changes to the positionsand/or orientations of the anatomical structure segments, for instancehow the anatomical structure segments can be repositioned with respectto each other in order to promote union between the anatomical structuresegments.

Three-Dimensional Overview of Imaging Scene of the Fixator

Another technique for improving accuracy and reliability of input valuesand resulting calculations is disclosed herein that provides athree-dimensional overview of an imaging scene of the fixator. Thethree-dimensional overview may be used to provide feedback and visualconfirmation to help ensure that the calculated positions andorientations of anatomical structures is reliable and correct. Varioustechniques for generating the three-dimensional overview will now bedescribed in detail with reference to FIGS. 18-25. In particular,referring now to FIG. 18 an example process for generating athree-dimensional overview of an imaging scene of a fixator includingrings and struts to correct a deformity of first and second anatomicalstructure segments will now be described in detail. The process of FIG.18 is initiated at operation 1810, at which first and second images,such as x-rays, of the fixator and the first and the second anatomicalstructure segments attached thereto are received, for example by acomputing system. As described above, the images can be captured usingone or more imaging sources, such as x-ray imagers and/or imagecapturing devices. As also described above, the first and the secondimages have respective first and second image planes at an angle withrespect to one another. The images may be received by a computingsystem, such as by scanning or otherwise electronically loading orcommunicating image data for the images to the computing system.

As set forth above, upon receiving the first and second images, framematching techniques may be employed in association with the first andsecond images, for example as described above with reference tooperations 314-334 of FIGS. 3A-3B, such as to determine positions andorientations of the anatomical structure segments, the fixator, theimaging sources, and other elements of the fixator imaging scene inthree-dimensional space. These techniques are described in detail aboveand are not repeated here. In some examples, upon completion of theframe matching process, for example including operations 314-334 (andoptionally the guided frame matching techniques of FIG. 14), athree-dimensional overview may be displayed to the user, such as willnow be described in detail. Specifically, at operation 1812 of FIG. 18,a three-dimensional graphical model is displayed, for example using oneor more graphical user interfaces of the computing system. Thethree-dimensional graphical model may include a first imagerepresentation that represents the first image and a second imagerepresentation that represents the second image. For example, referringnow to FIG. 19, it is seen that model 1900, which is a three-dimensionalgraphical model, includes an AP View image representation 1901-A thatrepresents an AP View image and a LAT View image representation 1901-Bthat represents a LAT View image. In the example of FIG. 19, the imagerepresentations 1901-A and 1901-B are graphical simulations of the firstand second images, which may be x-ray images. In some examples, however,the image representations 1901-A and 1901-B may include actual x-rayimages or other actual images that are captured from the imagingsources. It is noted that any, or all, of the contents shown in each ofFIGS. 19-25 may be displayed using one or more graphical user interfacesof a computing system.

In the model 1900, the image representations 1901-A and 1901-B may bedisplayed with respect to one another at the same angle as the anglebetween their respective image planes. For example, the image planes ofthe AP View image and the LAT View image may have an angle ofapproximately (but in some cases not exactly) ninety degrees withrespect to one another. Thus, in the model 1900, the AP View imagerepresentation 1901-A and the LAT View image representation 1901-B arepositioned with respect to one another at the same angle as theirrespective image planes (approximately ninety degrees). In someexamples, the software may calculate and display the actual anglebetween the image planes of the first and the second images as anumerical value.

In some examples, the model 1900 may display graphical representationsof respective locations of imaging sources of the first and the secondimages. In the example of FIG. 19, AP imaging source representation1911-A represents a location of an imaging source (e.g., x-ray imager,image capture device, etc.) of the AP View image, while LAT imagingsource representation 1911-B represents a location of an imaging source(e.g., x-ray imager, image capture device, etc.) of the LAT View image.The imaging source representations 1911-A and 1911-B may indicaterespective virtual locations corresponding to the first and the secondimaging sources. For example, upon performance of the frame matchingprocess, such as described above with reference to operations 314-334 ofFIGS. 3A-3B, the software may be capable of calculating positions andorientations of an entire three-dimensional scene including positionsand orientations of the anatomical structure segments, the fixator, andthe imaging sources with respect to one another. Thus, based on theframe matching process, the software may determine respective virtuallocations corresponding to the first and the second imaging sources.These virtual locations may reflect the distance, positions, and/ororientations of the imaging sources relative to the anatomical structuresegments and/or the fixator.

In the example of FIG. 19, a reference location 1913 is shown in boththe AP View image representation 1901-A and the LAT View imagerepresentation 1901-B. In this particular example, the referencelocation 1913 is a reference point, such as an endpoint, on a proximalanatomical structure segment 1915.

In some examples, a first virtual line may connect a first virtuallocation corresponding to the first imaging source to the referencelocation in the first image representation. In the example of FIG. 19, abeam 1912-A is shown as a graphical representation of this first virtualline that is displayed in the model 1900. As shown, the beam 1912-Aconnects a first virtual location corresponding to the first imagingsource (e.g., the location indicated by AP imaging source representation1911-A) to the reference location in the first image representation(e.g., the reference location 1913 in the AP View image representation1901-A).

Also, in some examples, a second virtual line may connect a secondvirtual location corresponding to the second imaging source to thereference location in the second image representation. In the example ofFIG. 19, a beam 1912-B is shown as a graphical representation of thissecond virtual line that is displayed in the model 1900. As shown, thebeam 1912-B connects a second virtual location corresponding to thesecond imaging source (e.g., the location indicated by LAT imagingsource representation 1911-B) to the reference location in the secondimage representation (e.g., the reference location 1913 in the LAT Viewimage representation 1901-B). Although beams 1912-A and 1912-B aredisplayed in model 1900, there is no requirement that the first andsecond virtual lines must be displayed in the model, and some models maynot display the beams 1912-A and 1912-B (or may display only portionsthereof).

At operation 1814, a first graphical representation associated with ashortest distance (e.g. intersection) between the first virtual line andthe second virtual line is displayed in the three-dimensional graphicalmodel. In the example of FIG. 19, beams 912-A and 1912-B intersect, anda point 1914 is shown at an intersection of the beams 1912-A and 1912-B,which is the shortest distance between beams 1912-A and 1912-B. Thus, inthis example, the point 1914 is a first graphical representation of theintersection between the first virtual line (represented by beam 1912-A)and the second virtual line (represented by beam 1912-B). In someexamples, as opposed to a point, the first graphical representation maysimply be an intersection between two graphical lines or beams or may beany other type of graphical representation. The first graphicalrepresentation (e.g., point 1914) represents a physical location of thereference location in three-dimensional space. Thus, in the example ofFIG. 19, the point 1914 represents a physical location of the endpointof proximal anatomical structure segment 1915 (which is the referencelocation 1913) in three-dimensional space.

In the example described above, the beams 1912-A (representing the firstvirtual line) and 1912-B (representing the second virtual line)intersect one another at the point 1914. In some examples, however, thefirst and second virtual lines described above may not actuallyintersect one another. In some examples, in these scenarios, as opposedto being displayed at the intersection point between the lines (since nointersection point exists), the first graphical representation mayinstead be displayed at a point that bisects (i.e., divides into twoequal halves) the shortest distance between the first virtual line andthe second virtual line. Specifically, a vector may connect respectivepoints on the first virtual line and the second virtual line at theshortest distance, and the first graphical representation may bedisplayed at the midpoint of the vector.

Operation 1814 is described above with respect to reference location1913 on proximal anatomical structure segment 1915. It is noted,however, that operation 1814 may be repeated for any number of otherreference locations. For example, FIG. 19 also shows a referencelocation 1923 in both the AP View image representation 1901-A and theLAT View image representation 1901-B. In this particular example, thereference location 1923 is a reference point, such as an endpoint, on adistal anatomical structure segment 1925. As shown, a beam 1922-Aconnects a first virtual location corresponding to the first imagingsource (e.g., the location indicated by AP imaging source representation1911-A) to the reference location 1923 in the AP View imagerepresentation 1901-A. Additionally, a beam 1922-B connects a secondvirtual location corresponding to the second imaging source (e.g., thelocation indicated by LAT imaging source representation 1911-B) to thereference location 1923 in the LAT View image representation 1901-B).Furthermore, a point 1924 is shown at an intersection of the beams1922-A and 1922-B. Thus, in this example, the point 1924 is a firstgraphical representation of the intersection between the beam 1922-A andthe beam 1922-B. The first graphical representation (e.g., point 1924)represents a physical position of the reference location inthree-dimensional space. Thus, in this example, the point 1924represents a physical location of the endpoint of distal anatomicalstructure segment 1925 (which is the reference location 1923) inthree-dimensional space.

At operation 1816, graphical representations of elements (e.g., rings,struts, etc.) of the fixator may be displayed, in the three-dimensionalgraphical model, at virtual locations that represent physical locationsof the elements of the fixator in the three-dimensional space. Forexample, referring now to FIG. 20, it is seen that a user may activate acontrol of the software, such as hexapod visible control 2001, whichcauses the model 1900 to display graphical representations of thefixator rings. The user may also deactivate the hexapod visible control2001, which causes the model 1900 to cease to display graphicalrepresentations of the fixator rings. As shown in FIG. 20, activation ofthe hexapod visible control 2001 causes a proximal ring graphicalrepresentation 2011 and a distal ring graphical representation 2012 tobe displayed in the model 1900. Additionally, referring now to FIG. 21,it is seen that a user may activate a control of the software, such asstruts visible control 2101, which causes the model 1900 to displaygraphical representations 2110 of the fixator struts. The user may alsodeactivate the struts visible control 2101, which causes the model 1900to cease to display graphical representations 2110 of the fixatorstruts. As shown, the graphical representations 2110 of the fixatorstruts may be color coded, such that different struts are shown indifferent colors, for example to match different colors of strutindicator buttons 611-A and 611-B of FIG. 6. In some examples, thephysical locations of the fixator elements in three-dimensional spacemay be determined by the software by performing the frame matchingtechniques described above (e.g., operations 314-334 of FIGS. 3A-3B).The software may then determine virtual locations within the model 1900that correspond to these physical locations, and the software maydisplay the graphical representations of the fixator elements at thedetermined virtual locations.

At operation 1818, graphical representations of the first and the secondanatomical structure segments may be displayed, in the three-dimensionalgraphical model, at virtual locations that represent physical locationsof the first and the second anatomical structure segments in thethree-dimensional space. For example, referring now to FIG. 21, it isseen that a user may activate a control of the software, such asproximal stick visible control 2102, which causes the model 1900 todisplay a graphical representation 2112 of the proximal anatomicalstructure segment 1915. The user may also deactivate the proximal stickvisible control 2102, which causes the model 1900 to cease to displaythe graphical representation 2112 of the proximal anatomical structuresegment 1915. As also shown in FIG. 21, a user may activate a control ofthe software, such as distal stick visible control 2103, which causesthe model 1900 to display a graphical representation 2113 of the distalanatomical structure segment 1925. The user may also deactivate thedistal stick visible control 2103, which causes the model 1900 to ceaseto display the graphical representation 2113 of the distal anatomicalstructure segment 1925. In some examples, the physical locations of theanatomical structure segments in three-dimensional space may bedetermined by the software by performing the frame matching techniquesdescribed above (e.g., operations 314-334 of FIGS. 3A-3B). The softwaremay then determine virtual locations within the model 1900 thatcorrespond to these physical locations, and the software may display thegraphical representations of the anatomical structure segments at thedetermined virtual locations.

Thus, as described above, the model 1900 may provide a three-dimensionalgraphical model that displays the first and the second images (orrepresentations thereof) in combination with graphical representationsof imaging sources, graphical representations of reference locations(e.g., endpoints of the first and the second anatomical structuresegments), graphical representations of virtual lines connecting virtuallocations corresponding to the imaging sources to the referencelocations and intersections thereof, graphical representations of thefixator elements (e.g., rings, struts, etc.), and graphicalrepresentations of the anatomical structure segments, thereby indicatingspatial relationships between these objects. Moreover, each of thegraphical representations may be displayed at virtual locations thatrepresent respective physical locations in three-dimensional space. Asshould be appreciated, the above described three-dimensional graphicalmodel may therefore provide visual feedback that allows the user tolocate points and other locations in three-dimensional space and reviewand confirm the correctness of the values calculated during the framematching process. For example, the user may confirm that referencelocations 1913 and 1923, such as endpoints of the anatomical structuresegments, are at correct locations in relation to other objects, such asthe fixator rings, fixator struts, and other locations on the anatomicalstructure segments. For example, in some cases, if the points 1914and/or 1924 were positioned at an incorrect location relative to thegraphical representations 2011 and 2012 of the fixator rings and/or thegraphical representations 2110 of the fixator sturts, then this would bean indication to the user that one or more calculations have not beenperformed correctly, for example due to user error in identifying inputvalues during the frame matching process. For example, if the points1914 and/or 1924 are positioned at non-sensical locations (e.g.,locations that collide with the fixator rings or struts, locationsoutward from the struts, location above an upper ring or below a lowerring, etc.), then this may be a clear indication of an error in theframe matching calculations. Upon determining such an error, the usermay choose to review and resubmit any or all input values providedduring the frame matching process and then reperform the calculations.

In some examples, the model 1900 may be zoomable, resizable androtatable, and otherwise manipulatable by the user. For example, theuser may zoom-in to enlarge portions of the model 1900 and/or zoom-outto increase the field of view. As another example, the user may panand/or rotate the model, such as in any combination of directions (e.g.,up, down, left, right, pitch, yaw, etc.). In some examples, the model1900 may be manipulated to be shown from various different perspectives,such as respective perspectives that correspond to the first and thesecond images. For example, referring now to FIG. 22, it is seen that auser may select a Show from AP button 2200 to cause the model 1900 to beshown from an AP view perspective corresponding to the AP View image.Additionally, referring now to FIG. 23, it is seen that a user mayselect a Show from LAT button 2300 to cause the model 1900 to be shownfrom a LAT view perspective corresponding to the LAT View image.Additionally, referring now to FIG. 24, it is seen that the model 1900may be rotated vertically relative to the perspectives shown in FIGS.19-23.

As described above, the first and second images corresponding to imagerepresentations 1901-A and 1901-B have image planes at an angle relativeto one another. For example, the AP View and LAT View imagescorresponding to image representations 1901-A and 1901-B may have imageplanes that are approximately orthogonal to one another. In some cases,however, these image planes may not be exactly orthogonal to oneanother. In some examples, when the image planes are not exactlyorthogonal, the software may display a modified second imagerepresentation that is truly orthogonal to the first imagerepresentation. The modified second image representation may represent amodified second image having an image plane that is truly orthogonal tothe image plane of the first image. For example, referring now to FIG.25, a modified LAT View image representation 2500 is shown in the model1900. The modified LAT View image representation 2500 is a modificationof LAT View image representation 1901-B. The modified LAT View imagerepresentation 2500 is truly orthogonal to the AP View imagerepresentation 1901-A within the model 1900. The modified LAT View imagerepresentation 2500 may represent a modified LAT View image having animage plane that is truly orthogonal to the image plane of the AP Viewimage. In some examples, a user may activate Ideal Plane Visible control2510 in order to cause the modified LAT View image representation 2500to be shown in the model 1900. The user may also deactivate the IdealPlane Visible control 2510 in order to cause the modified LAT View imagerepresentation 2500 to cease to be shown in the model 1900.

In some examples, the software may calculate the angle of the anatomicalstructure segments that would be displayed in the second image if thesecond image were truly orthogonal to the first image. The software maythen display, in the modified second image representation, a modifiedsecond image in which the anatomical structure segments are displayedwith the calculated angle with respect to one another. For example, thesoftware may calculate the angle of the anatomical structure segments1915 and 1925 that would be displayed in the LAT View image if the LATView image were truly orthogonal to the AP View image. The software maythen display, in the modified LAT View image representation 2500, amodified LAT View image in which the anatomical structure segments 1915and 1925 are displayed with the calculated angle with respect to oneanother. In some examples, the angles of the anatomical structuresegments in the modified LAT View image representation 2500 may bedetermined based on a knowledge of the positions and orientations of theanatomical structure segments in physical three-dimensional space, suchas may be determined by performing the frame matching techniquesdescribed above (e.g., operations 314-334 of FIGS. 3A-3B).

As should be appreciated, the ability to generate and display a modifiedsecond image representation (e.g., modified LAT View imagerepresentation 2500) may provide a number of benefits. For example, whena surgeon measures anatomical structure deformities on two-dimensionalimages, the assumption is that the images are orthogonal. Any deviationfrom the orthogonality of these images may lead to an inaccuratemeasurement, for example associated with angles in an anterior view(varus/valgus) and a lateral view (apex anterior/posterior) beginning tomix. This deviation may result in a residual deformity after thecorrection. Using frame matching techniques described above, thesoftware has knowledge of the positions and orientations of theanatomical structure segments in physical three-dimensional space. Thus,the software may calculate the angle of the anatomical structuresegments that would be displayed in the second image if the second imagewere truly orthogonal to the first image. In this way, the software maydemonstrate to the user how measured anatomical structure deformityvalues from a second image that is not truly orthogonal to the firstimage may be adjusted to corrected anatomical structure deformity valuesthat correspond to a modified second image that would be trulyorthogonal to the first image and provide a guidance to validate thecorrected values for use in the calculations described above. In someexamples, the user may be provided with an option to override thecorrected values (such as to use the measured values) if the user is notconfident with the corrected values.

It is noted that, while the model 1900 described above shows imagerepresentations for both the first and second images, athree-dimensional graphical model may also be provided that shows only asingle image representation for a single image. Such a single-imagemodel may be useful to provide feedback to the user at an earlier stageof the input process, such as when frame matching has only beenperformed on a single image, for example when a user has indicated strutlocations on the first image but not yet on the second image.Additionally, while the model 1900 described above shows points 1914 and1924 that represent reference locations of anatomical structures, athree-dimensional graphical model may also be provided that need notnecessarily include anatomical structure information. This type of modelmay also be useful to provide feedback to the user at an earlier stageof the input process, such as when user has indicated location offixator elements in the first and/or the second image-but has not yetindicated locations of anatomical structures in the first and/or thesecond image.

For example, for scenarios in which a user has indicated strut locationson a first image (but not yet for the second image) a single-image modelcould depict an image representation of the first image, an imagingsource representation for the first image, and graphical representationsof the frame (e.g., the rings and struts). Additionally, if the usersubsequently performs deformity planning and indicates the locations ofthe anatomical structure segments on the first image, then thesingle-image model could be updated to show planning elements (e.g.,anatomical structure reference points, anatomical structure centerlines) on the first image representation and to include beams thatconnect the imaging source representation for the first image tocorresponding points on the first image representation, in this wayshowing the user where they pass the fixator frame. Moreover, forscenarios in which identification of fixator element (e.g. strut)locations has been performed for both images but deformity planning(e.g., indication of anatomical structure locations) has not yet beenperformed, a model may be generated that shows image representations ofthe both images, imaging source representations for both images, andgraphical representations of the frame (e.g., the rings and struts),thereby allowing the user to review the relative orientation between theimages (i.e. angulation and rotation). Any or all of the above describedmodels may be rotated, zoomed and panned by the user similar to themodel 1900.

Changing of Distraction Rate

In some examples, during a course of a computer assisted ring fixatortreatment, a surgeon may desire a distraction rate change for a patient.This could be due to premature consolidation (correction is too slow),poor regenerate formation (correction is too fast), a strut swap at theclinic needing to be rescheduled, too much pain, or any other reason. Insome examples, in order to allow distraction rate to be changed in anefficient and reliable manner, an option may be provided within thesoftware for a surgeon to select a “change distraction rate” button orother control that will allow the distraction rate to be changed to anew distraction rate starting at a given day of the treatment plan andusing the new distraction rate for the remainder of the treatment plan.In some examples, this may allow the distraction rate to be changedduring the course of treatment (e.g., during an intermediate day withinthe treatment plan) using only one screen, one field, and one click toproduce a clinically relevant change to the patient treatment andpotentially reduce the patient time in the frame. In some examples, whenthe change distraction rate control is selected, the plan may bereopened by the software in a planning state, such as on a treatmentplan tab. In some examples, all other tabs (other than the treatmentplan tab) may be inactive. The user may then be able to edit either a“Number of Days” field (or other field that represents the treatmentplan duration) or a “Distraction at Reference Point” field. The user maythen select an “Update Adjustment Plan” control, and the software maygenerate a new treatment plan starting with the parameters from the dayat which the change distraction rate control was selected. The user maythen deliver the new plan to patient.

Example Computing Device

Referring to FIG. 26, a suitable computing device such as examplecomputing device 78 can be configured to perform any or all of thetechniques set forth above. It will be understood that the computingdevice 78 can include any appropriate device, examples of which includea desktop computing device, a server computing device, or a portablecomputing device, such as a laptop, tablet, or smart phone.

In an example configuration, the computing device 78 includes aprocessing portion 80, a memory portion 82, an input/output portion 84,and a user interface (UI) portion 86. It is emphasized that the blockdiagram depiction of the computing device 78 is exemplary and notintended to imply a specific implementation and/or configuration. Theprocessing portion 80, memory portion 82, input/output portion 84, anduser interface portion 86 can be coupled together to allowcommunications therebetween. As should be appreciated, any of the abovecomponents may be distributed across one or more separate devices and/orlocations.

In various embodiments, the input/output portion 84 includes a receiverof the computing device 78, a transmitter of the computing device 78, ora combination thereof. The input/output portion 84 is capable ofreceiving and/or providing information pertaining to communicate anetwork such as, for example, the Internet. As should be appreciated,transmit and receive functionality may also be provided by one or moredevices external to the computing device 78.

The processing portion 80 may include one or more processors. Dependingupon the exact configuration and type of processor, the memory portion82 can be volatile (such as some types of RAM), non-volatile (such asROM, flash memory, etc.), or a combination thereof. The computing device78 can include additional storage (e.g., removable storage and/ornon-removable storage) including, but not limited to, tape, flashmemory, smart cards, CD-ROM, digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, universal serial bus (USB)compatible memory, or any other medium which can be used to storeinformation and which can be accessed by the computing device 78.

The computing device 78 also can contain the user interface portion 86allowing a user to communicate with the computing device 78. The userinterface 86 can include inputs that provide the ability to control thecomputing device 78, via, for example, buttons, soft keys, a mouse,voice actuated controls, a touch screen, movement of the computingdevice 78, visual cues (e.g., moving a hand in front of a camera on thecomputing device 78), or the like. The user interface portion 86 canprovide outputs, including visual information (e.g., via a display),audio information (e.g., via speaker), mechanically (e.g., via avibrating mechanism), or a combination thereof. In variousconfigurations, the user interface portion 86 can include a display, oneor more graphical user interfaces, a touch screen, a keyboard, a mouse,an accelerometer, a motion detector, a speaker, a microphone, a camera,a tilt sensor, or any combination thereof. Thus, a computing systemincluding, for example, one or more computing devices 78 can include aprocessor, a display coupled to the processor, and a memory incommunication with the processor, one or more graphical user interfaces,and various other components. The memory can have stored thereininstructions that, upon execution by the processor, cause the computersystem to perform operations, such as the operations described above. Asused herein, the term computing system can refer to a system thatincludes one or more computing devices 78. For instance, the computingsystem can include one or more server computing devices that communicatewith one or more client computing devices.

While example embodiments of devices for executing the disclosedtechniques are described herein, the underlying concepts can be appliedto any computing device, processor, or system capable of communicatingand presenting information as described herein. The various techniquesdescribed herein can be implemented in connection with hardware orsoftware or, where appropriate, with a combination of both. Thus, themethods and apparatuses described herein can be implemented, or certainaspects or portions thereof, can take the form of program code (i.e.,instructions) embodied in tangible non-transitory storage media, such asfloppy diskettes, CD-ROMs, hard drives, or any other machine-readablestorage medium (computer-readable storage medium), wherein, when theprogram code is loaded into and executed by a machine, such as acomputer, the machine becomes an apparatus for performing the techniquesdescribed herein. In the case of program code execution on programmablecomputers, the computing device will generally include a processor, astorage medium readable by the processor (including volatile andnon-volatile memory and/or storage elements), at least one input device,and at least one output device, for instance a display. The display canbe configured to display visual information. The program(s) can beimplemented in assembly or machine language, if desired. The languagecan be a compiled or interpreted language, and combined with hardwareimplementations.

It should be appreciated that the orthopedic fixation with imageryanalysis techniques described herein provide not only for the use ofnon-orthogonal images, but also allow the use of overlapping images,images captured using different imaging techniques, images captured indifferent settings, and the like, thereby presenting a surgeon withgreater flexibility when compared with existing fixation and imagerytechniques.

The techniques described herein also can be practiced via communicationsembodied in the form of program code that is transmitted over sometransmission medium, such as over electrical wiring or cabling, throughfiber optics, or via any other form of transmission. When implemented ona general-purpose processor, the program code combines with theprocessor to provide a unique apparatus that operates to invoke thefunctionality described herein. Additionally, any storage techniquesused in connection with the techniques described herein can invariablybe a combination of hardware and software.

While the techniques described herein can be implemented and have beendescribed in connection with the various embodiments of the variousfigures, it is to be understood that other similar embodiments can beused or modifications and additions can be made to the describedembodiments without deviating therefrom. For example, it should beappreciated that the steps disclosed above can be performed in the orderset forth above, or in any other order as desired. Further, one skilledin the art will recognize that the techniques described in the presentapplication may apply to any environment, whether wired or wireless, andmay be applied to any number of such devices connected via acommunications network and interacting across the network. Therefore,the techniques described herein should not be limited to any singleembodiment, but rather should be construed in breadth and scope inaccordance with the appended claims.

What is claimed:
 1. A computer-implemented method for providing agraphical projection of a fixator including rings and struts to correcta deformity of first and second anatomical structure segmentscomprising: displaying, using one or more graphical user interfaces of acomputing system, first and second images of the first and the secondanatomical structure segments and the fixator attached thereto, thefirst and the second images having image planes at an angle with respectto one another; receiving, using the one or more graphical userinterfaces, first indications of first locations, within the firstimage, of a plurality of elements of the fixator; and overlaying, usingthe one or more graphical user interfaces, the graphical projection ofthe fixator on the second image, wherein the graphical projection of thefixator is rotated relative to the first locations based at least inpart on the angle.
 2. The computer-implemented method of claim 1,further comprising receiving, using the one or more graphical userinterfaces, second indications of second locations, within the secondimage, of the plurality of elements of the fixator.
 3. Thecomputer-implemented method of claim 2, wherein the overlaying of thegraphical projection of the fixator on the second image assists a userin identifying the second locations.
 4. The computer-implemented methodof claim 1, wherein the graphical projection of the fixator is movable,resizable and rotatable by a user.
 5. The computer-implemented method ofclaim 1, wherein the deformity is corrected using a treatment plan inwhich the distraction rate is changed during an intermediate day withinthe treatment plan.
 6. The computer-implemented method of claim 1,wherein the plurality of elements of the fixator include the struts. 7.The computer-implemented method of claim 1, wherein the graphicalprojection of the fixator includes graphical representations of thestruts.
 8. The computer-implemented method of claim 1, wherein thegraphical projection of the fixator includes graphical representationsof the rings.
 9. A computer-implemented method for generating athree-dimensional overview of an imaging scene of a fixator includingrings and struts to correct a deformity of first and second anatomicalstructure segments comprising: receiving, by a computing system, firstand second images of the fixator and the first and the second anatomicalstructure segments attached thereto, the first image captured from afirst imaging source and the second image captured from a second imagingsource, the first and the second images having respective first andsecond image planes at an angle with respect to one another; displaying,using one or more graphical user interfaces of the computing system, athree-dimensional graphical model that comprises a first imagerepresentation that represents the first image and a second imagerepresentation that represents the second image, wherein a first virtualline connects a first virtual location corresponding to the firstimaging source to a reference location in the first imagerepresentation, and wherein a second virtual line connects a secondvirtual location corresponding to the second imaging source to thereference location in the second image representation; and displaying,in the three-dimensional graphical model, a first graphicalrepresentation associated with a shortest distance between the firstvirtual line and the second virtual line, wherein the first graphicalrepresentation represents a physical location of the reference locationin three-dimensional space.
 10. The computer-implemented method of claim9, wherein the shortest distance between the first virtual line and thesecond virtual line is an intersection between the first virtual lineand the second virtual line, and wherein the first graphicalrepresentation is displayed at the intersection.
 11. Thecomputer-implemented method of claim 9, wherein the first virtual lineand the second virtual line do not intersect, and wherein the firstgraphical representation is displayed at a point that bisects theshortest distance.
 12. The computer-implemented method of claim 9,wherein the reference location is a reference point of the first or thesecond anatomical structure segment.
 13. The computer-implemented methodof claim 9, further comprising displaying, in the three-dimensionalgraphical model, graphical representations of the first virtual line andthe second virtual line.
 14. The computer-implemented method of claim 9,further comprising displaying, in the three-dimensional graphical model,graphical representations of the rings and the struts of the fixator atvirtual locations that represent physical locations of the rings and thestruts of the fixator in the three-dimensional space.
 15. Thecomputer-implemented method of claim 9, further comprising displaying,in the three-dimensional graphical model, graphical representations ofthe first and the second anatomical structure segments at virtuallocations that represent physical locations of the first and the secondanatomical structure segments in the three-dimensional space.
 16. Thecomputer-implemented method of claim 9, wherein the three-dimensionalgraphical model displays the first image representation and the secondimage representation at the angle with respect to one another.
 17. Thecomputer-implemented method of claim 9, further comprising, when thefirst and the second image planes are non-orthogonal, displaying amodified second image representation that is orthogonal to the firstimage representation.
 18. The computer-implemented method of claim 9,wherein the three-dimensional graphical model is zoomable and rotatableby a user.
 19. The computer-implemented method of claim 9, furthercomprising displaying, in the three-dimensional graphical model,graphical representations of the first virtual location corresponding tothe first imaging source and the second virtual location correspondingto the second imaging source.
 20. One or more non-transitorycomputer-readable storage media having stored thereon instructions that,upon execution by one or more computing devices, cause the one or morecomputing devices to perform operations for generating athree-dimensional overview of an imaging scene of a fixator includingrings and struts to correct a deformity of first and second anatomicalstructure segments comprising: receiving, by a computing system, firstand second images of the fixator and the first and the second anatomicalstructure segments attached thereto, the first image captured from afirst imaging source and the second image captured from a second imagingsource, the first and the second images having respective first andsecond image planes at an angle with respect to one another; displaying,using one or more graphical user interfaces of the computing system, athree-dimensional graphical model that comprises a first imagerepresentation that represents the first image and a second imagerepresentation that represents the second image, wherein a first virtualline connects a first virtual location corresponding to the firstimaging source to a reference location in the first imagerepresentation, and wherein a second virtual line connects a secondvirtual location corresponding to the second imaging source to thereference location in the second image representation; and displaying,in the three-dimensional graphical model, a first graphicalrepresentation associated with a shortest distance between the firstvirtual line and the second virtual line, wherein the first graphicalrepresentation represents a physical location of the reference locationin three-dimensional space.